Flaky test: WEAK-KEY-AND-VALUE-WEAKNESS fails intermittently on ubuntu-latest/bytecode

[dg1sbg]

WEAK-KEY-AND-VALUE-WEAKNESS (src/lisp/regression-tests/hash-tables0.lisp:147) fails intermittently in CI. Same commit, same job, opposite results:

run	commit	clasp/ubuntu-latest/bytecode/no/no
clasp-developers/clasp 30693073178	661438586	"Run regression tests" failed — Failures: WEAK-KEY-AND-VALUE-WEAKNESS, Successes: 1967
dg1sbg/clasp 30697205345	661438586 (identical)	"Run regression tests" passed — Successes: 1968

(The fork run's job still shows red, but for an unrelated reason: the runner received a shutdown signal during the later ANSI step. Its regression-test step succeeded.)

Only this OS/mode combination is affected. In the upstream run above, clasp/macos-latest/bytecode and clasp/ubuntu-latest/native both passed with the same commit.

Why it is fragile

(test weak-key-and-value-weakness
      (let ((table (make-hash-table :weakness :key-and-value)))
        (setf (gethash (list nil) table) :value
              (gethash :key table) (list nil)
              (gethash (list nil) table) (list nil)
              (gethash :key table) :value)
        (gctools:garbage-collect)
        (hash-table-count table))
      (1))

The test allocates three throwaway (list nil) values, triggers one GC, and asserts the count is exactly 1. Under Boehm's conservative stack scanning, a single stale word in a register or stack slot still referencing one of those temporaries keeps its entry alive and the count comes out 2. Whether that happens depends on register allocation and stack residue at the call site, so it is sensitive to unrelated codegen changes and to the state left behind by the ~20 suites that run before hash-tables0.

weak-key-weakness immediately above it has the same shape and presumably the same exposure.

Not reproducible locally

macOS arm64, both variants, all clean:

• 200/200 iterations of the test body in a fresh image
• 5/5 runs of TEST_SUITES=fastgf,clos,mop,hash-tables0
• full suites: boehm 1977, boehmprecise 1979, zero unexpected failures, including a from-scratch rebuild

Suggestion

Rather than asserting an exact count after one GC, either assert the surviving entry is present ((gethash :key table) → :value) and treat extra entries as tolerable, or retry the collect a bounded number of times before asserting. As written the test can redden any PR at random, which makes red CI easy to dismiss — the failure mode you least want on a project where CI is the stability gate.

Found while working on #1811 / #1812; unrelated to that change, which touches only CLOS class-identity guards and creates no new retention path (verified: the eql-specializer intern table is unchanged by hash-table use, and weak tables still collapse to 1 after 500 EQL dispatches with fully populated call histories).

[dg1sbg]

Superseded — the "does not reproduce on a big machine" conclusion below is wrong. See my follow-up comment for the correction.

In short: that negative result was statistically underpowered. The test runs once per suite run at roughly a 1.3% failure rate, so ~20 clean runs was the expected outcome of a measurement too weak to detect the effect, not evidence about the environment. It reproduces readily when exercised directly, the driver is stack depth (16.6% at 500 frames versus 1.3% shallow), and #1815's fix takes it from 165/1000 to 0/1000.

The frequency data in the original comment below still stands and is unaffected.

---

Original comment (frequency data valid; the reproduction conclusion is retracted)

Across five of my open PRs, WEAK-KEY-AND-VALUE-WEAKNESS was the sole failure in seven Linux jobs — #1817 (clasp/ubuntu native, clasp/ubuntu bytecode), #1812 (cando/ubuntu native), #1822 (cando/ubuntu native, passed on retrigger), #1826 (cando/ubuntu native), #1827 (clasp/ubuntu native, cando/ubuntu native). Every one shows Successes: 1962 with exactly this one failure and the normal 2 aborted compilation units. None of those PRs touch weak tables or the GC.

It is also not confined to ubuntu-latest/bytecode as the title suggests — six of the seven are native, and both clasp/ and cando/ variants are affected.

[dg1sbg]

Correcting my earlier comment: I said this "does not reproduce on a big machine". That was wrong — it reproduces readily, and I now have a reliable recipe plus a characterisation of what drives it.

What I got wrong

I reported ~20 clean full-suite runs, 13 more on an earlier occasion, and a 6-vs-6 A/B clean in both arms, and concluded the trigger needed conditions a GitHub runner has and my box does not.

That was a statistically underpowered negative dressed up as a finding. The test runs once per suite run at roughly a 1.3% failure rate, so 20 runs had about a 23% chance of showing a single failure. Zero failures was the expected outcome of a measurement too weak to detect the effect. The 6-vs-6 A/B was weaker still. Apologies — that pointed at runner hardware, which is the wrong place to look.

What actually drives it

Exercising the test's body directly rather than once per suite run, 2000 trials per condition, on a 16-core / 157 GB Ubuntu 24.04 box (LLVM 18, boehmprecise, --build-mode=bytecode):

stack depth at the call	failure rate
shallow	1.30 %
50 frames	0.05 %
500 frames	16.6 %

In every failure hash-table-count returned exactly 2 instead of 1 — a single (list nil) cons falsely retained, never more. That is an ordinary conservative-GC false retention: a stale word on the stack or in a register that happens to look like a pointer to the cons, keeping one weak entry alive across gctools:garbage-collect.

So the driver is stack depth, not core count or memory. That fits the observed spread across bytecode and native, clasp and cando — a stack-scanning artifact is indifferent to all of those — and it fits CI hitting it regularly, since each test runs beneath the suite runner's frames.

For anyone wanting to reproduce: call the test body under a few hundred frames of recursion and iterate a thousand times. It shows up in under a minute; no special environment needed.

#1815 fixes it

Same probe, 1000 trials per arm at depth 500 where the base rate is 16.5%:

form	depth	failures
current test	500	165 / 1000 (16.5 %)
#1815's form	500	0 / 1000
current test	shallow	1 / 1000
#1815's form	shallow	0 / 1000

Zero of a thousand against a 16.5% base rate; the 95% upper bound on the fixed arm is about 0.4%, so it removes at least a fortyfold factor and plausibly all of it.

That is what you would expect from the change: having the builder return before collecting pops the frame holding the dead temporaries, which is exactly the stale-reference source, and retrying the collection absorbs whatever residue survives that. A genuinely retained entry would still fail the test, so it does not weaken what is being asserted.

Not reopening

#1815 was closed by a maintainer, so I am leaving it closed rather than reopening it on my own initiative — there may be reasoning I am not aware of, and the shape of the fix (a retry loop in a test) is a fair thing to have reservations about.

Posting this only because the evidence now says the change works and the cost of the status quo is measurable: this single test accounted for nine job failures across six unrelated PRs in one day, which makes a red Linux job carry very little signal. If a retry loop is unwanted, the builder-returns half alone may be worth taking, since that addresses the mechanism directly — I am happy to measure that variant in isolation, or to test any other approach you would prefer.

[dg1sbg]

Opened #1830 with a fix, rather than reopening #1815 (which a maintainer closed, so I have left that decision alone).

#1830 differs from #1815 in one respect that may address whatever the reservation was: it drops the retry loop. #1815 did two things — have the builder return before collecting, and retry the collection up to ten times. Only the first removes the artifact; the second relaxes what the test asserts, from "collected in one GC" to "collected within ten".

Measuring them separately, 1000 trials per arm at stack depth 500:

form	on a 5.4 % build	on a 0.1 % build
current (build + collect in one frame)	54 / 1000	1 / 1000
builder returns first, one collect	0 / 1000	0 / 1000
builder + retry (#1815)	0 / 1000	0 / 1000

Builder-only is 0 failures in 2000 trials where about 55 would be expected, so the retry buys nothing measurable. #1830 therefore keeps the assertion exactly as it is today — one collection, exact count — and only stops the test from manufacturing the stale stack reference itself.

Non-vacuity is checked there too: an entry with both key and value pinned in globals still yields a count of 2 and still fails.

One correction to my numbers above: the 16.6 % figure is build-specific, not a constant. False retention depends on where stale words land in registers and stack slots, so code layout moves it — the same commit rebuilt gave 5.4 %, and another build 0.1 %. The range at depth 500 across three builds was 0.1 %–16.5 %. The qualitative result is unaffected.