Skip to content

Commit f819d69

Browse files
committed
feat: add the gate-standard local qualification campaign
Rebuild the qualification campaign fixture to the production gate standard (three trials per task x condition, explicit silent-incorrect-success and over-halt counts, and three uncertain-delivery fault conditions that return RECONCILIATION_REQUIRED with zero blind retries and zero replay dispatches). The harness reuses the bounded synthetic campaign pattern proven by the real-RDP multi-window campaign on a fully local substrate: Pillow-rendered pixels, a SQLite system of record plus an independent input-event surface, and the unmodified governed runtime (Recorder, compiler, Standard-profile run gate, qualification case authority, Replayer, independent effect verifiers). Every delivered input edge carries its own exact one-write effect contract, so duplicate or phantom edges fail independent verification. The counted summary refuses to write without every required counter, and the harness fails closed on a diverged condition contract. tests/ test_qualification_gate_campaign.py runs the 33-trial campaign once and asserts the full gate standard on its counted results; results.json retains the local acceptance run.
1 parent 7dd34db commit f819d69

7 files changed

Lines changed: 5908 additions & 0 deletions

File tree

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Qualification gate campaign (local, deterministic)
2+
3+
A counted qualification campaign rebuilt to the production gate standard
4+
(`AGENTS.md` §2): **three trials per task x condition**, an explicit counted
5+
summary that exposes `silent_incorrect_successes` and `over_halts`, and at
6+
least three expected uncertain-delivery fault conditions whose trials end in
7+
`RECONCILIATION_REQUIRED` — or in a contract-proven `VERIFIED` after
8+
uncertainty — with zero blind retries and zero replay dispatches.
9+
10+
It follows the structure and fail-closed conventions proven by the real-RDP
11+
multi-window campaign (`benchmark/rdp_multiapp`, PR #327): a machine-readable
12+
condition contract (`campaign.json`), a harness whose summary refuses to run
13+
without every required counter, per-trial fault evidence, and one accepted
14+
subset verdict for the whole matrix.
15+
16+
## Honest scope label
17+
18+
This is a **DETERMINISTIC LOCAL STAND-IN substrate**. It drives the unmodified
19+
governed runtime (Recorder -> compiler -> Standard-profile run gate ->
20+
qualification case authority -> Replayer -> independent SQLite effect
21+
verifiers) against Pillow-rendered pixels and a local SQLite system of record.
22+
23+
It does NOT exercise FreeRDP or Citrix transports, real window managers,
24+
hosted browsers, or multi-monitor topologies. Those remain bound to the hosted
25+
campaigns (`benchmark/rdp_multiapp` on hosted Linux with Docker + FreeRDP;
26+
the Citrix ICA/HDX lane's real-protocol acceptance). Hosted-only parts are
27+
exactly the transport-level faults: window/session identity, focus theft
28+
across windows, and codec behavior are qualified by those campaigns, not this
29+
one. What this campaign qualifies is the substrate-independent gate behavior:
30+
verified healthy effects, safe halts before consequential input on identity /
31+
ambiguity / render faults, and exactly-once uncertain-delivery reconciliation.
32+
33+
## Conditions and expectations
34+
35+
See `campaign.json`. Summary of the implemented matrix (3 trials each):
36+
37+
| Condition | Expectation |
38+
| --- | --- |
39+
| `healthy` | verified |
40+
| `row_reordered` | verified |
41+
| `moderate_display_drift` | verified_or_safe_halt |
42+
| `severe_display_drift` | safe_halt |
43+
| `duplicate_save_control` | safe_halt |
44+
| `partial_render` | safe_halt |
45+
| `wrong_record_before_write` | safe_halt |
46+
| `stale_identity_before_write` | safe_halt |
47+
| `uncertain_delivery_write_lost` | reconciliation_required |
48+
| `uncertain_delivery_write_kept_timeout` | verified_after_uncertainty_or_reconciliation_required |
49+
| `uncertain_delivery_oracle_unreachable` | reconciliation_required |
50+
51+
Every delivered input edge carries its own exact one-write effect contract on
52+
an independent persisted event surface (`input_events`) beside the qualified
53+
business surface (`records`), so duplicate or phantom edges fail independent
54+
verification rather than relying on screen state.
55+
56+
## Run
57+
58+
```bash
59+
python3 benchmark/qualification_gate/run_campaign.py \
60+
--output benchmark/qualification_gate/results.json
61+
```
62+
63+
No Docker, no network, no browser, no model calls. The harness fails closed:
64+
a missing required metric, a diverged condition contract, any wrong-record
65+
write, duplicate effect, silent incorrect success, over-halt, blind retry,
66+
replay dispatch, or model call makes the campaign refuse acceptance
67+
(non-zero exit). `tests/test_qualification_gate_campaign.py` runs the same
68+
campaign once and asserts the full gate standard on the counted results.
69+
70+
## Recorded result
71+
72+
`results.json` retains the counted 33-trial local run (2026-08-22, exact
73+
branch head): 12 verified outcomes, 15 safe halts, 18 reconciliation-required
74+
outcomes, and zeros for silent incorrect successes, over-halts, wrong-record
75+
writes, duplicate effects, model calls, blind retries, and replay dispatches.
Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
{
2+
"schema_version": "openadapt.qualification-gate-campaign.v1",
3+
"task": "Select one synthetic patient record, enter a triage note parameter, and prove both the exact persisted record and every delivered input edge on independent read-only SQLite surfaces through a deterministic pixel fixture.",
4+
"trials_per_condition": 3,
5+
"required_metrics": [
6+
"verified_outcomes",
7+
"safe_halts",
8+
"reconciliation_required_outcomes",
9+
"silent_incorrect_successes",
10+
"over_halts",
11+
"wrong_record_writes",
12+
"duplicate_effects",
13+
"model_calls",
14+
"blind_retries",
15+
"replay_dispatches"
16+
],
17+
"minimum_uncertain_delivery_conditions": 3,
18+
"conditions": [
19+
{
20+
"id": "healthy",
21+
"expect": "verified",
22+
"fault": "none",
23+
"oracle": ["records", "input_events"]
24+
},
25+
{
26+
"id": "row_reordered",
27+
"expect": "verified",
28+
"fault": "The target record row renders at a different vertical position.",
29+
"oracle": ["records", "input_events"]
30+
},
31+
{
32+
"id": "moderate_display_drift",
33+
"expect": "verified_or_safe_halt",
34+
"fault": "A bounded scale, contrast, and compression change keeps text legible.",
35+
"oracle": ["records", "input_events"]
36+
},
37+
{
38+
"id": "severe_display_drift",
39+
"expect": "safe_halt",
40+
"fault": "Heavy downscale, inverted luminance, posterization, blur, and recompression remove reliable target detail.",
41+
"oracle": ["records", "input_events"]
42+
},
43+
{
44+
"id": "duplicate_save_control",
45+
"expect": "safe_halt",
46+
"fault": "Two competing Save controls render inside the recorded target region, defeating template match and retained-evidence disambiguation.",
47+
"oracle": ["records", "input_events"]
48+
},
49+
{
50+
"id": "partial_render",
51+
"expect": "safe_halt",
52+
"fault": "The Save control is absent when its action is due.",
53+
"oracle": ["records", "input_events"]
54+
},
55+
{
56+
"id": "wrong_record_before_write",
57+
"expect": "safe_halt",
58+
"fault": "The active record changes to a different patient before the consequential click.",
59+
"oracle": ["records", "input_events"]
60+
},
61+
{
62+
"id": "stale_identity_before_write",
63+
"expect": "safe_halt",
64+
"fault": "The identity band is replaced by a loading placeholder before the consequential click.",
65+
"oracle": ["records", "input_events"]
66+
},
67+
{
68+
"id": "uncertain_delivery_write_lost",
69+
"expect": "reconciliation_required",
70+
"fault": "Delivery certainty is lost after the input edge and the application rolls its commit back.",
71+
"oracle": ["records", "input_events"]
72+
},
73+
{
74+
"id": "uncertain_delivery_write_kept_timeout",
75+
"expect": "verified_after_uncertainty_or_reconciliation_required",
76+
"fault": "Delivery certainty is lost after the input edge and the write persists; only the contract may confirm it.",
77+
"oracle": ["records", "input_events"]
78+
},
79+
{
80+
"id": "uncertain_delivery_oracle_unreachable",
81+
"expect": "reconciliation_required",
82+
"fault": "Delivery certainty is lost after the input edge and the system of record becomes unreachable before proof.",
83+
"oracle": ["records", "input_events"]
84+
}
85+
]
86+
}

0 commit comments

Comments
 (0)