Ancestor-subzone Zarr layout: one chunk is one subtree

[asinghvi17]

Stack position: 2 of 4 — main ← #62 claude/dgg-source-subtree ← #63 claude/subzone-store ← #64 claude/copdem-production ← #65 claude/perf-ladder (draft)

This is a stacked PR chain. Each PR's base is the previous branch, so each diff shows only its own commits. Do not squash-merge out of order; merge bottom-up and retarget children after each merge.

---

What

A second Zarr store layout, layout = :subzones: a 2-D ancestor-subzone array in which dim 1 is the subzone position 1:capacity within one subtree and dim 2 is the level-La ancestor, chunked (capacity, 1) — so one Zarr chunk is one level-La subtree. A level-L cell set becomes columns of 7^(L−La) cells per level-La ancestor.

Plus the verbs to fill it incrementally (subzonestore, dggwrite!) and a DiskArrays facade that reads it back as the one-dimensional cell axis it stands for.

Why the existing 1-D layout could not do this

Zarr chunks are uniform by format, so a chunk grid that follows the tree exactly is not expressible along a 1-D cell axis: an IGeo7 pentagon's subtree holds p(d) = (5·7^d + 1)/6 cells where a hexagon's holds 7^d, and no single chunk length lands on both. The subzone layout buys tree-aligned chunking by spending a dimension on it.

There is a second, harder motivation recorded in regrid-notes/2026-08-20-copdem-zarr-spike.md §3: _cellaxis/_rawids materializes the raw-id vector for every encoding (:ranges, :dense, :implicit alike). A global IGeo7 L12 store is 138,412,872,012 cells — 1.107 TB for the id vector alone, on a 751 GB machine, plus ~1.4e11 rawid/convert calls. This layout never materializes a level-L id vector at all: mapping in both directions is ancestor / cellposition / descendant_range, O(level) digit arithmetic wherever has_sorted_subtrees holds, so a store of 10^12 cells is described by three integers.

Three properties it buys

1. Sparsity is free. An ancestor nobody wrote is a chunk that was never stored; it reads back as fill at zero bytes. No mask, no index, no compaction step.
2. A column is one chunk is one file, and a column write rewrites nothing shared — not the attributes, not the consolidated metadata, not a manifest. Disjoint columns may therefore be written from as many tasks as you like with no coordination. The suite asserts exactly that: one new file per column write, and every file that was there before untouched to the mtime.
3. The reader gets the tree's own chunking back, as DiskArrays.IrregularChunks of the real column lengths — 7^d for a hexagon ancestor, p(d) for the twelve pentagons — with padding dropped. Anything that reads by chunk now reads whole subtrees. A read inside one column is one chunk read; one spanning columns is one per column. Opening is O(1) even at L13.

The four commits

• 39b1094 — the arithmetic and vocabulary, src/io/subzones.jl (624 lines), Zarr-free and system-generic. SubzoneLayout and the mapping both ways (subzoneindex, positionindex, columnpositions); the run planner that turns a cube's cell axis into whole columns and refuses a partially covered one; its inverse subzone_cellvector; the attribute block a store carries. subzonestore/dggwrite! join dggread/dggwrite as stubs here.
• 05f8af0 — the Zarr half, ext/DiscreteGlobalGridsZarrExt/subzones.jl (626 lines). subzonestore(dest, system, level; …) creates the group, its (capacity, ncolumns) arrays chunked one column each, and its attributes, once; subzonestore(dest) reopens. dggwrite!(store, ancestor, values; var) fills columns afterwards. dggwrite(dest, cube; layout = :subzones, ancestor_level = k) is those two calls and nothing else, so the incremental path is not a second implementation of the one-shot path. Reading: dggread recognizes the layout from its own attribute before the conventions are asked, and returns a Cells dimension over SubzoneCellArray <: DiskArrays.AbstractDiskArray{T,1}. DiskArrays becomes a direct dependency (an extension may only load what the package declares; Zarr brings it in either way).
• c784f23 — the store-IO reference section: what the layout buys, what it costs (a column is written whole), where each verb lives, alongside the 1-D layout it sits beside.
• 86f3174 — covers the stack form of the one-shot write, which goes through the same subzonestore/dggwrite! pair, one layer per element type, and was previously unasserted.

On-disk vocabulary

One group attribute dggs with everything the layout adds nested under subzone_layout: layout: "ancestor_subzone", version: 1, ancestor_level, ancestor_dimension, ancestor_count, ancestor_order: "ascending_id", subzone_count, padding: "trailing_fill", padding_fill_value, chunk_shape. ancestor_count/subzone_count are checked against the grid's own arithmetic on read, not believed.

Deliberately no zarr_conventions declaration. This is not the one-dimensional layout that convention describes, and claiming to be it would send a convention-aware reader down a path that cannot open the store; a foreign reader instead gets a clean "no convention detected". Column order inside a subtree is what OGC API-DGGS calls sub-zone order.

layout is orthogonal to encoding (store shape vs coordinate shape) and is not a CellEncoding.

Pentagons — the one structural caveat

p(d) = (5·7^d + 1)/6. Confirmed by descendant_range at La=2, L=4: exactly 12 of 492 columns are short — 41 cells against capacity 49 — and they are exactly the digit-0 chains from the twelve root cells. Padding is asserted: 41 real values then 8 NaN, and the axis returns 41 cells. values must be columnlength(layout, column) long; handing 7^d to a pentagon column raises an ArgumentError rather than silently overwriting padding.

Sizing

Chunk size is capacity × sizeof(T) — the only thing the layout constrains La by.

use	La	d	chunk cells	chunk (f32)	columns
GLO-90 → L12 (production)	5	7	823,543	3.14 MiB	168,072 global
GLO-90 → L12	6	6	117,649	470 KB	1,176,492
GLO-30 → L13	6	7	—	3.3 MB	1,176,492
GLO-30 → L13	7	6	—	470 KB	8,235,432 files

The optional ancestor_cell_ids coordinate is on by default and costs 9.4 MB at La=6, written once at creation; this package never reads it back. capacity is measured once at creation (one descendant_range per level-La cell — a fraction of a second even at 1.2 M columns); pass capacity = 7^d to skip that pass.

Deliberately not built

Multi-dimensional layers (a time/band cube is refused by name), written-column discovery or an index, a chunk cache in the wrapper (use DiskArrays.cache), remote writes, and writing through the read view (refused, and points at dggwrite!). Coverage must be ancestor-snapped: a cube covering part of a subtree raises DGGSFormatError(check = :incomplete_subtree).

Tests

test/io/subzones.jl 110 assertions (no Zarr) plus test/io/subzone_store.jl 86 assertions (self-skipping when Zarr is absent) → io suite 937 assertions / 0 failures, 196 of them new. Whole package 987,456 pass / 17 broken / 0 fail.

Notes

Full record: regrid-notes/2026-08-20-copdem-zarr-spike.md §14. The La choice this layout is sized against is in regrid-notes/2026-08-20-la-choice.md (chunking is bitwise-neutral: La=5 vs La=6 vs La=7 over the same region gave max |diff| 0.000e+00 m, 218,491/218,491 bitwise equal — so La can be revisited later without invalidating output).

🤖 Generated with Claude Code

https://claude.ai/code/session_01AC7rbi9eqZypWseMjBGSJ2

[asinghvi17]

wall of text. also, this is a bit of a hack so should go in a separate file, once zarrv3 with the right stuff lands we'll just get rid of it

[asinghvi17]

Moved out of store-io.md into its own page, docs/src/api/subzone-layout.md, which opens with an "Interim" admonition saying it exists only because Zarr chunks are uniform and goes away once variable chunk sizes land; the prose is roughly half its old length. The implementation was already in its own files (src/io/subzones.jl and ext/DiscreteGlobalGridsZarrExt/subzones.jl) — the former's header now leads with the same interim note. 13ebe23

[asinghvi17]

WALL OF TEXT

[asinghvi17]

subzone_attrs is down to eight lines: what the attribute object holds, why there is no zarr_conventions key, and what coordinate is for. Also cut the file header and the other new walls in this diff (the ext header, SubzoneCellArray, the subzonestore stub). 13ebe23

[asinghvi17]

comments left

[asinghvi17]

comments left