Add a regression test for truncated-then-full sample-cache runs#1291
Open
shipbehaves wants to merge 1 commit into
Open
Add a regression test for truncated-then-full sample-cache runs#1291shipbehaves wants to merge 1 commit into
shipbehaves wants to merge 1 commit into
Conversation
Running with --max-samples N and then again without it must not serve back only the truncated cache. The cached decorator already handles this on the current default branch: it reprocesses the docs that are not cached yet and returns the full requested set while reusing the already-cached ones. Lock that behavior with a DummyModel-based test (CPU-only, no model download) that asserts the full set comes back, cached docs are reused, and only the missing doc is reprocessed. Verified red-green by temporarily suppressing the reprocessing. Refs huggingface#1040
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1040, where running with
--max-samples 10and then without it served back the cached truncated data.On current main the reported symptom no longer reproduces. Verified two ways:
@cachedwrapper appends not-yet-cached docs todocs_not_cachedand reloads the full requested set)lighteval accelerateonarc:challengewith--max-samples 10, then again without it, sharing the cache dir; the first run cached exactly 10 samples and the second run computed and returned all 1172This PR adds a regression test that locks the per-doc contract so the original bug cannot silently come back: after a truncated run, a full run must return the full set, reuse the cached docs, and reprocess only the missing ones. The test uses
DummyModel(no model download, CPU-only, runs in a few seconds) and was verified red-green by temporarily suppressing the reprocessing inget_samples_to_process_and_cache.One observation from the end-to-end check, possibly worth a separate issue: the two runs landed in different task-hash cache slots.
_get_task_hashhashesLightevalTaskConfig.__str__(lite=True), and that string includes the solver's function repr with a memory address ([<function multiple_choice.<locals>.solve at 0x...>]), which changes every process. So cache slots are currently never shared across CLI invocations for tasks with a solver, which quietly defeats cross-run reuse. Happy to open a small follow-up PR to stabilize the hash input (e.g. use qualified names instead of default function reprs) if useful.Closes #1040