Icechunk Multiscales Demo

Example workflow for adding multi-resolution overviews (image pyramids) to an existing Icechunk dataset, following GeoZarr conventions.

Relevant specs:

• multiscales - pyramid layout metadata
• spatial: - spatial coordinate information
• proj: - coordinate reference system

What This Does

This tool generates overview pyramids for large geospatial Icechunk datasets stored in Arraylake. Overviews are downsampled versions of your data at progressively lower resolutions, enabling efficient visualization—when zoomed out, the viewer loads lower-resolution data instead of the full dataset.

Metadata handling:

• Assumes the native dataset already has GeoZarr metadata (proj:, spatial:)
• Adds GeoZarr metadata to each newly created overview group (proj:, spatial:)
• Creates/updates the multiscales attribute on the root group to include the new overview level

Why Two Stages?

Overview generation is split into two stages to handle large, sparse datasets efficiently:

Stage 1: Initialize (--initialize)

Creates the overview group with empty arrays filled with fill values across the entire spatial extent. This "grids out" the overview structure without computing any actual data.

Why separate this? For global datasets where data may be sparse (e.g., land-only data over mostly ocean), we don't want to iterate over every tile during processing. By initializing first, we establish the complete grid structure, and then selectively fill only the tiles that contain data.

Stage 2: Process (default)

Actually computes and writes the downsampled data. This stage:

• Optionally filters by bounding box: Only processes tiles intersecting with a specified --data-bbox. For sparse data, this avoids wasting compute on empty regions.
• Parallelizes across workers: Each job processes one (tile × variable × time step) combination, distributed across Coiled workers.
• Commits in batches: Work is committed in configurable batches (--batch-size) so that:
  • Progress isn't lost if the process fails partway through
  • Memory usage stays bounded on the Coiled cluster
  • The Icechunk repo doesn't accumulate too many uncommitted changes

Prerequisites

• Python 3.12+
• An Arraylake account and repository with an existing Icechunk dataset
  • The source dataset should be GeoZarr-compliant with spatial and proj metadata
• A Coiled account for distributed processing
• Environment variables:
  • ARRAYLAKE_TOKEN: Your Arraylake API token

Important: The overview generation script assumes a specific dimension order:

• 3D data: (time, y, x) — e.g., (time, latitude, longitude)
• 4D data: (time, y, x, band) — e.g., (time, latitude, longitude, band)

Even single time-step datasets must have a time dimension of size 1.

Installation

# Install dependencies with uv
uv venv
source .venv/bin/activate
uv pip install -r requirements.txt

# Or with pip
pip install -r requirements.txt

Usage

See USAGE.md for the full usage guide, including CLI reference, group structure options, resampling methods, and customization.

Step 0: Export your Arraylake token

This will be passed to the Coiled workers to read and write to your dataset.

export ARRAYLAKE_TOKEN='ema_<my-token>'

Step 1: Initialize the Overview Level

python src/create_overviews.py \
    --arraylake-repo-name your-org/your-repo \
    --downsample-factor 4 \
    --initialize

This creates the 4x/ group with empty arrays. Run once per overview level.

Step 2: Process Tiles

python src/create_overviews.py \
    --arraylake-repo-name your-org/your-repo \
    --downsample-factor 4 \
    --data-bbox -125 24 -66 50  # Optional: only process tiles in this region

Without --data-bbox, all tiles are processed.

Comparing Overview Levels

Use the included Jupyter notebook to visualize results:

jupyter notebook compare_overviews.ipynb

Known Limitations / TODOs

• Add Icechunk manifest-based filtering to detect empty chunks without requiring --data-bbox
• Support for processing multiple overview levels in a single run