Write and use consolidated metadata

[Bisaloo]

Related to: fix #461

Description

In line of what anndata python is doing, this PR:

• always writes consolidated metadata when writing anndata to Zarr.
• uses the metadata when reading (it's created on the fly if it doesn't exist on disk) to avoid multiple reading of multiple JSON files.

Eventually, in a follow up PR, this can also be used to add reading from S3.

Checklist

Before review

• Update and regenerate man pages
• Add/update tests
• Add/update examples in vignettes
• Pass CI checks

Before merge

• Update NEWS
• Bump devel version

[Bisaloo]

I'll stop here for a first review & merge if this seems good.

The 2nd stage, which will bring reading from S3 and further performance improvement, will require changing the signature of some helper function, so I prefer to do it in a dedicated PR.

For example, read_zarr_categorical() would now have ordered in its signature so it can be passed directly from the consolidated metadata, and all read_zarr_attributes() calls can be avoided (or even eliminated) from individual low-level helpers.

[github-actions]

Bencher Report

Project	anndataR
Branch	consolidated-metadata
Testbed	ubuntu-latest

⚠️ WARNING: Truncated view!

The full continuous benchmarking report exceeds the maximum length allowed on this platform.

🚨 9 Alerts

🐰 View full continuous benchmarking report in Bencher

[lazappi]

I think this looks good. Does there need to be any checks for the Zarr version?

[lazappi]

I'm not sure if we want to bump this yet (I've lost track of how we were managing versions, @rcannood might have a plan)

[Bisaloo]

Does there need to be any checks for the Zarr version?

From Rarr's side, no. Consolidated metadata is generated, and would be beneficial, for both v2 and v3. I don't know if the anndata spec is strict about this but my view is that even if anndata python doesn't generate it / use it for v2, they are not likely to be hindered by it either.