Store the largest IGeo7 oracle as a zstd frame - #27
Open
asinghvi17 wants to merge 1 commit into
Open
Conversation
`dggrid_true_res5_centers.txt` is 7.6 MB, the largest tracked file in the repo and about 3.4 MB of a clone once git's zlib has had it. It is 168,072 rows — every cell at level 5, which is 86% of the 196,080 rows this oracle carries. Compressed it is 2.7 MB, on disk and in the pack both. Nothing about the oracle changes: `zstd -dc` on the new file reproduces the old one byte for byte, same git hash, so this is a re-encoding and not a regenerated dump. The suite still reports the same 5.913e-09 deg worst centroid error at every level, against the 1e-8 the test allows. Compression rather than precision, deliberately. Trimming the coordinates to ten decimals would save less (6.0 MB, 2.6 MB packed) and spend real margin: the observed error is already 6e-9 against a 1e-8 threshold, so there is under a factor of two to give away, and once trimmed it cannot be recovered without DGGRID. `load_true_centers` now takes whichever of `.txt` or `.txt.zst` is present. Only level 5 is converted — level 4 is 1.1 MB and level 3 is 155 KB, small enough that staying greppable is worth more than the bytes — and dispatching on the extension means converting another later needs no further code change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test/systems/IGeo7/vectors/dggrid_true_res5_centers.txtis the largest trackedfile in the repo: 7.6 MB, 168,072 rows, every cell at IGeo7 level 5. That is 86%
of the 196,080 rows the oracle carries, and roughly 3.4 MB of a clone once git's
zlib has had it.
Stored as a zstd frame it is 2.7 MB, on disk and in the pack both.
The oracle does not change
zstd -dcon the new file reproduces the old one byte for byte — same git hash,533fb6faa74fff8674d9441278cc3e6ed30145f2. This is a re-encoding, not aregenerated dump, so there is no possibility of the digits having quietly moved.
The suite confirms it from the other side: testset 4 still reports a worst
centroid error of
5.913e-09deg at every level 1–5, against the1e-8itallows, with zero decode mismatches.
Why compression and not precision
Trimming the coordinates to ten decimal places was the alternative. It saves
less — 6.0 MB raw, 2.6 MB packed — and it spends margin this test does not have
to spare: the observed error is already 6e-9 against a 1e-8 threshold, under a
factor of two, and once trimmed the digits cannot be recovered without rerunning
DGGRID.
The loader
load_true_centerstakes whichever ofdggrid_true_res{r}_centers.txtor….txt.zstis on disk. Only level 5 is converted here: level 4 is 1.1 MB andlevel 3 is 155 KB, small enough that staying greppable is worth more than the
bytes. Dispatching on the extension means converting another one later needs no
change to this function.
CodecZstdbecomes a declared test dependency. It was already in the manifesttransitively (via
Healpix), so it costs no new download.🤖 Generated with Claude Code