Elo results on live elo page differ from results on raw statistic page for some tests

[vdbergh]

I have no time to debug this right now but I am recording this for further reference

The Elo upper bound for https://tests.stockfishchess.org/tests/live_elo/6a39895b3036e45021aeb368 is 1000 whereas on the corresponding raw statistics page https://tests.stockfishchess.org/tests/stats/6a39895b3036e45021aeb368 it is 147.8685 (which is much more reasonable).

It is also interesting that the 5 pentanomial LLR estimates on https://tests.stockfishchess.org/tests/stats/6a39895b3036e45021aeb368 are wildly different, but this must be because these estimates were derived under the assumption that Elo differences are small, an assumption that does not hold in this case.

[ppigazzini]

Baseline behavior

Both pages ultimately call the same estimator, sprt.sprt.analytics, which
returns elo = lower_cb(0.5) and ci = [lower_cb(p/2), lower_cb(1 - p/2)]
(server/fishtest/stats/sprt.py:115-124). The disagreement comes entirely from
how each page parameterizes the SPRT object before calling analytics.

Live Elo path (_build_live_elo_context, views.py:2701-2711):

• Reads elo_model from the run, default BayesElo
  (views.py:2703); real SPRT tests store normalized
  (server/fishtest/schemas.py:774,
  server/fishtest/templates/tests_run.html.j2:6).
• Calls SPRT_elo(results, ..., elo_model="normalized").
• Inside SPRT_elo, the normalized branch keeps the bounds in normalized
  units and builds sprt.sprt(elo0, elo1, elo_model="normalized")
  (stat_util.py:135). For a standard STC test the bounds are
  elo0 = 0.0, elo1 = 2.0, so delta = elo1 - elo0 = 2.0.

Stats path (_build_sprt_pentanomial, template_helpers.py:525-602):

• Always builds sprt_module.sprt(elo0=lelo0, elo1=lelo1) with the default
  elo_model="logistic" (template_helpers.py:542-547), where lelo0/lelo1
  are the bounds converted to logistic Elo through the pentanomial score
  (_build_sprt_model_params, template_helpers.py:404-420). For the same test
  the logistic bounds are about 0.0 and 1.036, so delta is about 1.036.

Both then call analytics() with the same default p = 0.05 on the same
regularized pentanomial. Regularization is not the difference: results_to_pdf
regularizes internally (server/fishtest/stats/LLRcalc.py:223-227), so passing
a raw or a pre-regularized pentanomial yields the same pdf.

Root cause

lower_cb inverts outcome_prob(elo) with scipy.optimize.brentq over a
search window whose width is scaled by the SPRT bounds in the model's own units
(server/fishtest/stats/sprt.py:83-113):

avg_elo = (self.elo0 + self.elo1) / 2
delta   = self.elo1 - self.elo0
N       = 30
elo0 = max(avg_elo - N * delta, -1000)
elo1 = min(avg_elo + N * delta,  1000)

Two facts make this fragile:

1. outcome_prob always maps Elo to a score with the logistic law
   s = LLRcalc.L_(elo) (sprt.py:72-81), independent of self.elo_model.
   So the root being sought lives on the logistic Elo axis, but the window step
   size delta is in the caller's model units. Normalized delta = 2.0 and
   logistic delta = 1.036 therefore expand the window on different schedules.

2. When the observed Elo is far outside [elo0, elo1] (the regime the reporter
   flags), the GSPRT LLR is clamped in set_state (sprt.py:63-70) and
   outcome_prob(elo) is no longer a monotone CDF. It is approximately 1.0
   almost everywhere with a narrow notch dipping toward 0. lower_cb
   silently assumes monotonicity, so whether brentq brackets a sign change or
   falls through to the +/-1000 clamp depends on whether a window edge happens
   to land inside the notch.

Verified reproduction on a synthetic strongly-passing test
(pentanomial = [1, 20, 300, 900, 779], standard STC bounds [0.0, 2.0],
alpha = beta = 0.05), running the repository modules directly:

outcome_prob(elo) scan:
  op(-1000)=1.00000  op(0)=1.00000  op(200)=1.00000
  op( 245)=0.51391   op(300)=0.01782 op(500)=0.99533
  op( 800)=1.00000   op(1000)=1.00000        <-- non-monotone notch near 245..300

lower_cb(0.5), logistic bounds (delta=1.036):
  N=240 window=[-248.08, 249.12] straddles the notch -> 245.473   (finite)

lower_cb(0.5), normalized bounds (delta=2.0):
  N=240 window=[-479.00, 481.00] steps over the notch -> no sign change
  ... expands to [-1000, 1000] -> terminal clamp -> 1000

s0, s1, T, llr, and outcome_prob are bit-identical between the two
parameterizations; only the lower_cb window schedule differs. That single
difference decides 1000 (live Elo, normalized) versus a finite value (stats,
logistic). This exactly matches issue #2571: live Elo shows the clamp while
stats shows the finite 147.8685.

The reporter's own diagnosis is correct: both numbers are unreliable here
because the small-Elo Brownian approximation does not hold. The finite stats
value is not "more correct"; it is a spurious-but-finite root that the logistic
window happened to bracket.

Evidence map

Concern	Location
Live Elo context builder	server/fishtest/views.py:2695-2746
Live Elo elo_model default BayesElo	server/fishtest/views.py:2703
ci_upper_raw / ci_upper sent to template	server/fishtest/views.py:2719-2727
SPRT_elo bound conversion by model	server/fishtest/stats/stat_util.py:120-154
SPRT_elo drops the clamped flag	server/fishtest/stats/stat_util.py:152
lower_cb window scaled by model-unit delta	server/fishtest/stats/sprt.py:83-113
+/-1000 window clamp and terminal fallback	server/fishtest/stats/sprt.py:92-110
outcome_prob hardcodes logistic L_(elo)	server/fishtest/stats/sprt.py:72-81
analytics builds elo/ci from lower_cb	server/fishtest/stats/sprt.py:115-124
Stats builds logistic SPRT, ignores clamped	server/fishtest/http/template_helpers.py:542-592
Stats pentanomial "Elo" row (decimals=4)	server/fishtest/http/template_helpers.py:673-683
Default SPRT model normalized	server/fishtest/schemas.py:774
Live Elo renders Elo [ci_lower,ci_upper]	server/fishtest/templates/tests_live_elo.html.j2:92
ci_upper_raw also drives the gauge scale	server/fishtest/templates/tests_live_elo.html.j2:40-41

Audit: done right / meh / wrong

Done right:

• A single estimator (sprt.sprt.analytics) backs both pages, so the fix has a
  single natural home.
• set_state already detects the degenerate regime and sets self.clamped
  when the LLR is out of range (sprt.py:63-64).
• The stats page keeps the model-free delta-method CI available on the
  non-SPRT branch via get_elo (template_helpers.py:919-936,
  stat_util.py:52-68); this is the kind of "reasonable" estimate the reporter
  expects.

Meh:

• The two pages disagree on which Elo model to report in. Live Elo reports in
  the run's native model (normalized), stats always reports logistic. Even away
  from the degenerate regime this makes the numbers coincide only by accident
  of the standard bounds; it is not a documented, intentional contract.
• lower_cb mixes a domain expansion heuristic (N *= 2) with a hard
  physical clamp (+/-1000) and a silent monotonicity assumption, with no
  signal to the caller when it degrades.

Wrong:

• lower_cb returns the raw clamp value 1000 (or -1000) as if it were a
  real Elo when the inversion is ill-posed. That value is meaningless and, on
  live Elo, also rescales the gauge.
• The clamped flag that exists to mark exactly this situation is discarded by
  both call sites: SPRT_elo deletes it (stat_util.py:152) and
  _build_sprt_pentanomial never reads a5["clamped"]. The system computes
  the warning and throws it away.
• The user-facing result depends on an internal numeric detail (model-unit
  window step) that has no statistical meaning.

Options

Option A - Unify the estimator call (consistency only)

Route both pages through one helper that parameterizes the SPRT object
identically (same model, same bounds), so they never disagree.

• Backend: smallest change; removes the reported inconsistency at the call
  sites. Does not make lower_cb well-posed, so both pages would then show the
  same unreliable number (either both 1000 or both the spurious finite value).
• Frontend: no template change required.
• DevOps/SRE: low risk, easy to test.
• Technical writer: still leaves an unexplained 1000 on screen; support
  questions persist.

Option B - Make lower_cb model-invariant and well-posed (root cause)

Search on the logistic Elo axis that outcome_prob actually uses, with a
window independent of the caller's model units, and define behavior when the
inversion is ill-posed: detect non-monotone outcome_prob / no finite
crossing and return the existing clamped signal instead of a raw +/-1000.

• Backend: fixes the real defect once, for every consumer of analytics
  (live Elo, stats, elo_results, API). Requires care because analytics
  feeds many surfaces; must be validated so non-degenerate results are
  unchanged to full precision.
• Frontend: consumes a clean out-of-range signal to render >elo1 or
  "out of SPRT range" instead of 1000, and to bound the gauge.
• DevOps/SRE: medium risk; central statistics code. Needs a focused test set
  and a diff against current output on a corpus of finished runs.
• Technical writer: enables an honest, stable message for the degenerate case.

Option C - Degrade to a model-free estimate when assumptions break (UX + honesty)

When the run is out of SPRT range (LLR clamped / outcome_prob non-monotone),
show the delta-method Elo and CI from get_elo (the same estimator already
used on the non-SPRT branch), clearly labeled as an out-of-range fallback, on
both pages.

• Backend: reuses get_elo; the fallback is well understood and always finite.
• Frontend: one labeled state; the number matches what users intuitively expect
  (the reporter called this class of value "much more reasonable").
• DevOps/SRE: low added risk once the detection from Option B exists.
• Technical writer: clean contract - "inside SPRT range: dynamic CI;
  out of range: sample CI, flagged."

Option D - Documentation only

Explain the discrepancy without code change.

• Rejected: leaves a wrong on-screen value and an internal inconsistency that
  generates repeated reports.

Recommendation

Adopt Option B as the core fix, layered with A and C:

1. B first: make lower_cb search on the logistic Elo axis with a
   model-invariant window, and return the clamped / out-of-range signal
   instead of a bare +/-1000 when no valid crossing exists. This removes the
   normalized-versus-logistic divergence at the root, because the window
   schedule stops depending on the caller's model units.
2. A falls out of B: with a model-invariant inversion, both pages agree by
   construction. Keep the reported Elo model explicit and consistent between
   the two pages while doing so.
3. C for honesty: when out of range, both pages present the model-free
   delta-method CI from get_elo, clearly labeled, rather than a dynamic CI
   the assumptions do not support. Stop discarding the clamped flag so the UI
   can trigger this state.

Rationale: the disagreement and the 1000 are the same defect - an ill-posed
inversion whose outcome leaks an internal numeric detail to users. Fixing the
inversion (B) is the only change that makes both pages correct rather than
merely equal, and it benefits every consumer of analytics. A and C are small
once B exists and turn a confusing failure into a defined, documented state.

[vdbergh]

Thanks!! I need a bit of time to digest this :)

[ppigazzini]

I opened a PR running on DEV, please review at your convenient time :)
#2572

https://dfts-0.pigazzini.it/tests/live_elo/6a39895b3036e45021aeb368
https://dfts-0.pigazzini.it/tests/stats/6a39895b3036e45021aeb368

[dubslow]

Nice work.