CompactedEncoding: mixed-level cell axes in Zarr stores

[asinghvi17]

Stacked on #29 (it needs the MultiOrderVector/MultiOrderLookup container from that branch and the store-IO layer from main). Retarget to main once #29 merges.

Implements the mixed-level ("compacted", in zarr-conventions/dggs vocabulary) store layout that #29's harmonization pass left as a named refusal:

• CompactedEncoding in the encoding registry as "compacted". Two aligned columns in container order: cell_ids (the grid's own id type, the declared coordinate) and cell_levels (Int8, on its own dimension so it never reads as a data variable). Container order is required on read, not restored — a sort would silently misalign the value arrays (:compacted_axis_order).
• Write: encoding = :auto selects compacted for a MultiOrderLookup axis; an explicitly requested single-level encoding keeps the refusal naming expand. chunks = :auto descends coarse-ancestor runs on the container; the persisted manifest is keyed by reference-level interval start/stop and its marker records level: null plus a new reference_level field.
• Read: the axis decodes to a validated MultiOrderVector and dggread returns a DimArray over a MultiOrderLookup; selectors work against the read-back axis. A store declaring both compression: "compacted" and a non-null refinement_level is refused (:level_with_compacted).

Two behavioral decisions to review:

1. Compacted stores get no xdggs stamp — xdggs attrs can only describe a dense single-level coordinate, and dual-stamping would make the two conventions disagree on read.
2. refinement_level: null + compression: "compacted" is the one legal spelling; a non-null level alongside compacted is refused rather than trusted.

Numbers: io suite 784/784 offline, 799/799 with DGG_IO_NETWORK_TESTS=1 (live Pori stores still read); full suite 988,086 pass / 17 pre-existing broken / 0 fail. Round trips are value- and axis-identical on HEALPix (radix 4) and IGeo7 (radix 7, unsigned Z7 ids), including a coarsen output.

Deferred: missing/fill-value support (pre-existing gap in every encoding — it is why the WorldClim tutorial has no write demo); a packed single-column id form (the registry and storedlookup seam are ready for it); Zarr v3.

🤖 Generated with Claude Code

[asinghvi17]

Updated on top of the refreshed claude/moc-storage (merge commit f6a23ce, not a rebase).

Conflicts and how they were resolved

• src/DiscreteGlobalGrids.jl — the store-io registries (CONVENTION_REGISTRY, DEFAULT_WRITE_CONVENTIONS, ENCODING_REGISTRY, GRID_REFERENCE) became public rather than exported upstream; kept that and added CompactedEncoding to the encoding export line.
• src/io/api.jl — merged the compacted paragraph with mains :ranges/:implicit sentence, and widened the "cell dimension carries" line to name MultiOrderLookup alongside the single-level lookups.

Beyond the conflicts, three references followed MultiOrderVectors move from Fallbacks to Engine: the encoding's import, _writemixed's reference_level call, and the read test's covering_position.

Fast eager path, end to end through a compacted store

coarsen a level-5 IGEO7 field (168,072 leaves → 26,496 stored cells at levels 3–5), dggwrite it (compression: "compacted", refinement_level: null, cell_ids + cell_levels), dggread it back as a MultiOrderLookup, then regrid a 1° global raster onto that read-back axis eagerly.

_asspace resolves it to DGGSpace(168072 cells, 72 chunks at level 1) and plan_regrid(..., lazy = false) dispatches to wholeblock(::Conservative, ...) in conservative.jl, not the generic COO fallback — a temporary counter in both paths reports 2 specialized calls and 0 _fillcoo! calls. DirectPlan / WeightBlock{SparseMatrixCSC{Float64,Int64},Vector{Float64}}, 487,176 nonzeros. Plan 0.59 s, regrid 1.02 s, -t 8, loadavg 22.3.

Suites (Julia 1.12, -t 8): root 988,915 pass / 17 broken / 0 fail in 16m08s, loadavg 20.8 → 24.9. The io suite alone is 792/792.

[asinghvi17]

Codex review of the compacted layout — 5 findings, 3 fixed, 1 accepted-not-fixed, 1 rebutted

Codex (GPT-5.6 sol, high reasoning) reviewed origin/claude/moc-storage...HEAD and ran the feature end to end: coarsen → dggwrite → on-disk inspection → dggread → index → regrid, plus malformed-store probes. Fixes in 87a7065.

1. compression: "compacted" under refinement_level: null is not conforming — [major] — ACCEPTED, not fixed here.
Real, and broader than reported. The spec's prose rule is unconditional, not HEALPix-only:

If refinement_level is null, compression MUST be "none".

so both radices are affected, not just healpix. The healpix oneOf in schema.json additionally pins refinement_level: null to a ^[a-z_]*uniq$ indexing scheme, and we write nested. I re-ran Codex's validation against the live v1 schema and confirm it.

Not fixable by a metadata tweak: the conforming shape for mixed-level healpix is a *uniq id — level folded into the id, no level column at all — which is a different on-disk format and a different reader (describedsystem currently refuses any scheme but nested). That is a format decision, not a review fix, and it is cheaper to make before this lands than after. What I did do is stop the docs claiming the convention blesses this layout; CompactedEncoding's docstring and store-io.md now say plainly that it is an extension a conforming reader will reject. Flagging for a call on whether to go nuniq before merge.

2. cell_levels is not discoverably tied to cell_ids — [major] — FIXED.
Nothing in the group attrs named the levels array, and its separate _ARRAY_DIMENSIONS (deliberate — otherwise it reads back as a data variable) actively says "unrelated". The store now names it itself, in the one place an independent reader looks:

"dggs": {..., "coordinate": "cell_ids", "refinement_levels": "cell_levels", "refinement_level": null}

additionalProperties: true on the dggs object, so this costs no conformance we still had. Kept the separate array dimension.

3. Out-of-range stored level → InexactError — [minor] — FIXED.
Int(cell_levels[k]) on a wild unsigned threw before the level check could run. Membership is now tested before narrowing, so it lands on :invalid_stored_level like every other bad level.

4. encoding = CompactedEncoding() bypassed eligibility — [ergonomics] — FIXED.
The instance form skipped write_eligible and fell through to a self-contradictory unsupported_encoding ("compacted names an encoding it has no write path for" — on the encoding we just added a write path for). _encoding(::CellEncoding, ...) now applies the same check the keyword does; both forms answer :not_write_eligible with the MultiOrderLookup/expand pointer. Added a write_eligible(::CellEncoding, ...) = true fallback so a downstream encoding that states no restriction still reaches the by-name refusal.

5. Empty compacted axis readable but not writable — [minor] — REBUTTED.
Not compacted-specific and not introduced here — the refusal is the pre-existing rule for every encoding. Measured on this branch:

F5 empty SINGLE-level dense write => ArgumentError: dggwrite has nothing to write: the cell axis is empty.
F5 empty MIXED-level write        => ArgumentError: dggwrite has nothing to write: the cell axis is empty.

Identical. If empty stores should be writable that is a separate change across all four encodings.

What Codex could not break

Missing/short/non-integer cell_levels, mismatched cell_ids vs data length, a dense store relabelled compacted, duplicates, ancestor/descendant overlap, unsorted pairs, degenerate single-level MultiOrderLookup, lazy write→read→write — each gave a named DGGSFormatError, and round-trip ids/levels/values/ordering, At/Contains, expand and regrid were all exact.

Suites

test/io 796/796 (baseline 792 + 4 new asserts). Root suite Julia 1.12 -t 8: 988,919 pass / 17 broken / 0 fail (branch baseline 988,915/17/0, +4 = the new asserts). loadavg 55.8 → 50.4 on a shared 64-core box.