Give each schedule worker its own claim variable - #73
Conversation
`GuidedSchedule taper` failed intermittently in CI — PR #70 on 1.12-macos, PR #71 on 1.12-ubuntu, the same assertion, neither PR anywhere near the scheduler. The cursor was not the problem. The testset's tail-loop above already binds `b` in the testset scope: while (b = claim!(s)) !== nothing last = b end so the `b` in the concurrent loop below it is not a fresh local. It is that same binding, and because eight `Threads.@spawn` closures assign to it, lowering hoists it into one shared `Core.Box` they all capture. The optimized closure is explicit about the consequence: setfield!(box, :contents, %2) # my claim %5 = (%2 === nothing) # loop test reads the fresh value ... # seen[w], bounds-checked %32 = getfield(box, :contents) # append! re-reads the SHARED box append!(%26, %32) Between the store and the re-read, any of the other seven workers can store its own batch. The loser then appends the winner's range: one batch counted twice, its own dropped. The total stays 5000 and the ends stay clean, which is exactly the failure CI printed — the damage is in the elided middle. A worker that stores `nothing` on its way out can also hand a live worker `append!(seen[w], nothing)`. Reproduced by widening that store-to-load window with `--compile=min`: 13-16 mismatches per 200 trials, every one of them `ndup = ngap = 8`, plus task exceptions from the `nothing` case. 2200 trials of the fixed shape across 2, 4, 8 and 16 threads: zero. Hoisting the loop into `drain!` makes `b` a local of one call, so each task owns its own; the spawn body no longer assigns anything. `claim!` itself is correct and unchanged. Its CAS loop was stressed at 32.4M claimed positions over a grid of n, workers and maxbatch at 8 and 24 threads with tasks oversubscribed against workers: every run covered `1:n` exactly once. Production `runchunks` lowers with zero `Core.Box` — its `batch` is bound only inside the spawned closure — so the completed 66,228-column run was never exposed to this. Report the miscover in parts, too, so the next one names itself: count, duplicated, uncovered, stray, rather than two collections that print identically at both ends. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019DuGKTmvs5B4EynwZddKMg
The failure inventory is worse than it looked: 4 occurrences, not 2I went back to the raw job logs. The
Both 1.11 jobs report The "1.11-ubuntu is red on main for 8 allocation-law asserts" belief is stale — So this one test was the only thing red on #70 and #71, on every job that was |
GuidedSchedule taperhas been failing intermittently in CI and blocking theG-stack: PR #70 failed
1.12 - macos-latestand passed ubuntu, PR #71 failed1.12 - ubuntu-latestand passed macos, same assertion attest/scripts/copdem_policy.jl:107, and neither PR touched the scheduler.It is the test, not
claim!The tail-loop directly above already binds
bin the testset scope:so the
bin the concurrent loop below it is not a fresh local — it is thatsame binding. Eight
Threads.@spawnclosures assign to it, so lowering hoistsit into a single
Core.Boxthat all eight capture. Confirmed both ways:Meta.loweremitsCore.Boxforbonly when that tail loop is present, andat runtime four tasks writing four distinct values all read back one value.
The optimized closure shows what that costs:
Between the store and the re-read, any of the other seven workers can store its
own batch. The loser appends the winner's range: one batch counted twice, its
own dropped. Length stays 5000 and both ends stay clean — precisely the shape
CI printed, with the damage in the elided middle. A worker storing
nothingonits way out can also hand a live worker
append!(seen[w], nothing).Reproducer
The store-to-load window is a few instructions, which is why this never fired
on the dev box (five clean local runs, 8000 trials at 2/4/8/16 threads, 5M
claims at
n = 2_000_000: nothing). Widening it with--compile=minfires itreadily:
Every mismatch was
count = 5000, duplicated = 8, uncovered = 8— one batchdoubled, one batch dropped.
claim!is correct, and production was never exposedclaim!is unchanged. Its CAS loop was stressed at 32.4M claimed positionsover a grid of
n(0, 1, 2, 7, 8, 9, 63, 64, 1000, 100 000),workers(1, 2, 8, 24) and
maxbatch(1, 3, 8, 64), at 8 and 24 threads with tasksoversubscribed against workers: every run covered
1:nexactly once and leftthe cursor at
n + 1.Production
runchunksinscripts/copdem_production.jllowers with zeroCore.Box— itsbatchis bound only inside the spawned closure, and theclosure captures only read-only state (
chunks, order, sched, dem, srcspace, sys7, store, layout, prefetcher, config, log, p, w). The completed66,228-column run's "0 skipped, complete store" stands; no column was double-
regridded or dropped by this. The only other
claim!loops are single-threaded.The fix
Hoist the claim loop into
drain!, sobis a local of one call and the spawnbody assigns nothing. Only
sremains boxed, and it is read-only while thetasks run.
The assertion is also split into named parts —
count,duplicated,uncovered,stray— so the next miscover says what was wrong instead ofprinting two collections that are identical at both ends.
Not on the stack
Cut from
origin/claude/perf-ladder, independent of #70–#72, so it can mergewithout waiting on them.