GENERATED by
scripts/validate_claims.py --reportfromclaims.yaml. Do not edit by hand — edit the registry and regenerate.
- Generated at: committed registry state (regenerate: scripts/validate_claims.py --report)
- JUnit pass check: not embedded in this generated registry view (required CI jobs enforce pass evidence)
- Structure gate:
python scripts/validate_claims.py --check --structure-only(a claim whose tier outranks its strongest backing evidence fails CI). - Pass gates: required
testande2e-browserjobs supply their own JUnit files; an absent, all-skipped, or failed supported evidence file fails that required job. - Scoped validating refreshes name each selected evidence file; each selected file must pass on its declared substrate. Unselected validating evidence is not represented as checked.
What this harness does and does not do. It makes each public maturity claim a function of automated evidence: a supported claim must be backed by a test file that has a real passing case in its required default CI job; a validating claim must be grounded in a REAL opt-in / infra-gated proof or a field test, and is never presented as supported. It does not replace workflow- and deployment-specific acceptance: application controls, identity rules, effect oracles, and live transport conditions remain bound to their counted evidence.
- Bound to required CI pass evidence (11):
web-supported,deterministic-zero-model-replay,effect-verification-silent-writes,identity-gate-halt-armed,identifier-crop-compile-emission,halt-teach-promote,typed-business-decisions-supported,reviewed-judgment-cases-supported,qualified-remote-decision-v2-supported,mockmed-benchmark-ci-reproducible,substrate-runtime-validation-supported - Being validated — opt-in / infra-gated or field (8):
effect-verifier-kit,windows-desktop-validating,macos-native-validating,linux-native-validating,rdp-validating,desktop-recording-validating,citrix-pixel-validating,openemr-field-benchmark - Roadmap / research (1):
win32-window-replay-roadmap
Web (browser) workflows are supported today: record a GUI workflow once, then replay it deterministically and locally.
- Surfaces: README.md, website, docs
- Strongest evidence strength: supported (tier is
supported)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/e2e/test_record_compile_replay.py |
test | ci (required PR gate (e2e-browser)) | supported | Records the MockMed browser demo once, compiles it, and replays it under baseline + theme/move/rename drift and parameter substitution through the headless-browser Backend. |
tests/test_mockmed.py |
test | ci (required PR gate (test)) | supported | The reference browser demo app and its drift screens render deterministically (no CSS transitions), so replay is repeatable. |
tests/test_replayer.py |
test | ci (required PR gate (test)) | supported | The deterministic replayer resolves steps, substitutes parameters, enforces postconditions and the risk gate — no model in the loop. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | Three real Chromium CDP-attach trials record and compile the same synthetic workflow, exclude password values before persistence, preserve CSS-pixel frame/coordinate alignment, and detach without closing the external browser. The same live campaign covers same-task replacement, declared open and closed shadow hosts, contenteditable click metadata, and same-document URL/title reflection through history.replaceState. A live case records and compiles actions across viewport and monitor-scale changes. It binds each event to its exact frame dimensions. A separate live case refuses an action that overlaps the transition. Unit cases refuse remote endpoints, cross-origin selectors and navigation, iframe events, invalid viewport evidence, and ambiguous same-origin tabs. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | A live Chromium recording submits a same-origin GET form. The document that submit reaches is a fresh closure that holds no value to match against, so Flow withholds its URL and its title and says so, and the recorded surface is stamped before publish. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | A server that answers a GET submit with a 302 to /results/<value> puts a declared secret in a path segment, where no parameter name identifies it. Flow withholds the whole URL for every document after the one that first held the value, so the literal never reaches events.jsonl. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | A lowercase secret typed one character at a time into a page whose URL, title, and button ID share those characters leaves the URL, the title, the DOM selector, and the accessible name exact. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | A password that begins with a common English word leaves that word exact in the page URL, the page title, the clicked row's identity characters, the accessible name, and an unrelated button ID. Matching uses only the value the field holds at that moment, so an intermediate keystroke prefix never becomes a matching value. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | Identity evidence that holds a declared value is WITHHELD with a stated reason, never rewritten. Replay compares identity evidence against the live page, so a rewritten copy would compare against characters the page never showed, invisibly. No placeholder string appears anywhere in the recorded evidence. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | A page that replaces its input element on every keystroke and writes the value into its URL and title can end up showing a version the field no longer holds. Flow withholds that reflected text whole -- an origin-only URL and an empty title -- instead of matching it against the current value, and keeps the clicked element's identity evidence exact. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | A single-page application route change through history.pushState does not build a new document, so the closure that held the declared value is the one being sampled and its URL is still reported exactly. The cross-document rule that withholds a later document's URL therefore does not cost this evidence. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | A scanner input that writes the badge into the page URL and title and then clears its own field holds nothing at any moment Flow samples. Flow arms the document's secret boundary from the input event itself, so both channels are checked and both are withheld. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | A second scan into the same cleared field does not displace the first badge while the first badge is still shown in the URL. A value the next one does not continue was taken by the page, not edited away by the operator, so it is promoted into the withhold-only set. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | A second declared field holding a value does not re-expose the first field's consumed value. The last-value test is per element, so the same URL is withheld before and after the second field is filled. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | A DOM selector is identity evidence and uses the same value set as the accessible name and the clicked-row identity. An element id built from a value carried in an inbound declared parameter is withheld with a reason, not emitted verbatim. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | A query parameter whose value Flow cannot prove predates the moment the document first held a declared value loses only ITS value. Every parameter name survives, the path stays exact, and the drop is recorded with its reason. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | A value that no parameter name identifies -- here written into a path segment -- is caught by detection rather than structure: Flow withholds the whole URL, marks it, and warns the operator that the application placed a declared secret into its own URL. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | A single-page wizard that removes its declared field and renders the value into a summary row cannot leak it into the clicked row's identity: the value the field held at a commit point is retained for the single purpose of WITHHOLDING identity text, never to rewrite it and never for the URL or the title. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | Flow never reads the visible text of a bound secret field, because that text is the value. The resulting missing accessible name is reported as WITHHELD with a reason and counted, never left as a silent null, while a control field beside it returns its name. |
tests/test_browser_attach.py |
test | ci (required PR gate (test)) | supported | No browser event carries a URL or a title. The in-page listeners run in the capture phase, before the page's own handlers, so anything they read describes the state before the action. Flow samples reflected evidence from Python at the settled boundary instead. |
Caveats (honest limits):
- "Supported" is scoped to the reference headless-browser backend in this registry. Desktop and remote-display workflows use the separately scoped acceptance and code-qualified claims below.
- The full record->compile->replay browser suite runs in the required e2e-browser PR gate and repeats in the weekly compatibility matrix.
- Existing-session attachment is Chromium-only and loopback-only. It requires a dedicated browser process started with remote debugging. It does not claim support for the Capture Chrome extension prototype or direct extension replay.
Replay is deterministic, local, and $0 — zero model calls on either the record or the replay side.
- Surfaces: README.md, website, docs
- Strongest evidence strength: supported (tier is
supported)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/e2e/test_record_compile_replay.py |
test | ci (required PR gate (e2e-browser)) | supported | Baseline replay x3 resolves every step on the template rung with 0 heals and 0 model calls; healed bundles replay clean afterward. |
tests/test_replayer.py |
test | ci (required PR gate (test)) | supported | The replayer is a deterministic resolution ladder; the VLM rung is a fallback, not on the default path — the core runtime is model-free. |
benchmark/BENCHMARK.md |
benchmark | artifact (doc/benchmark) | roadmap | CI-reproducible MockMed run artifact: 100/100 compiled replays at 4.9s p50 with zero model calls, measured 2026-07-08 on Flow 0.1.0 (pre-v0.2.0 source build); field/artifact reference. |
Caveats (honest limits):
- A VLM rung exists as the lowest fallback of the ladder; it is not invoked on a clean deterministic replay, and never on the record side.
Effect verification against the system of record catches silent wrong writes the screen-only oracle misses (5 of 7 transactional fault classes).
- Surfaces: README.md, docs
- Strongest evidence strength: supported (tier is
supported)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_effect_verifier.py |
test | ci (required PR gate (test)) | supported | The REST/FHIR/document-hash verifiers confirm exactly-one write and refute duplicate / missing / partial / collateral-loss writes. |
tests/test_silent_wrong_action_benchmark.py |
test | ci (required PR gate (test)) | supported | Each silent-wrong-action class is silent under the screen-only oracle and is CAUGHT once effects are declared and verified. |
tests/test_effect_fhir.py |
test | ci (required PR gate (test)) | supported | The FHIR effect verifier's read/confirm/refute/indeterminate contract (the live-OpenEMR smoke is a separate opt-in function in this file). |
benchmark/fault_model/FAULT_MODEL.md |
benchmark | artifact (doc/benchmark) | roadmap | The 7-class transactional fault-model study: 5/7 classes silently mishandled by screen-only, all 5 halt through the real replayer. |
Caveats (honest limits):
- Verification requires the step to DECLARE typed
effectsAND a configured EffectVerifier. Without a real system-of-record verifier, on-screen read-back is SAME-SURFACE and not independent.
The effect-verifier kit makes declaring and wiring effect verification a reviewed deployment-config exercise instead of bespoke per-deployment code: declarative REST / FHIR / read-only-SQL / file-arrival / document-hash verifier configs with secret-isolated auth references and explicit run-parameter binding, a cross-substrate exactly-one-NEW-record duplicate guard, per-consequential-step effect-coverage lint/certify gates, and typed reconciliation tasks on halt. A deployment with multiple reviewed read boundaries selects the strongest evidence tier before actuation, retains that exact selection through durable resume, and never downgrades to a weaker verifier after an action.
- Surfaces: docs
- Strongest evidence strength: supported (tier is
validating)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_effect_sql.py |
test | ci (required PR gate (test)) | supported | The read-only SQL verifier's whitelist (mutation/stacking/comment smuggling refused at construction) and full verdict contract, plus the exact table-delta audit promoted from the Frappe Lending reference matrix — against LOCAL sqlite fixtures only. |
tests/test_effect_file_arrival.py |
test | ci (required PR gate (test)) | supported | File-arrival verdicts (pattern / size>0 / mtime freshness / content probe / duplicate export) against real temp directories, and the SFTP path against an in-memory fake paramiko-shaped transport. |
tests/test_effect_kit_config.py |
test | ci (required PR gate (test)) | supported | The declarative construction path: env-var auth references fail loud when unset, run-parameter references resolve at build (and refuse to construct unresolved), and pre-kit configs build byte-identically. |
tests/test_effect_verifier_candidates.py |
test | ci (required PR gate (test)) | supported | Required CI constructs and preflights all reviewed candidates before input, pins the strongest verifier and its pre-state, preserves the binding through durable resume, and refuses an unavailable selected verifier rather than falling back after an action. |
tests/test_effect_coverage_lint.py |
test | ci (required PR gate (test)) | supported | lint warns per consequential step lacking an effect contract and reports coverage %; certify fails the same gap only when the policy sets require_effects_for_irreversible (warn-vs-fail is policy-configurable). |
tests/test_effect_reconciliation.py |
test | ci (required PR gate (test)) | supported | Every escalated verification failure emits a typed ReconciliationTask carrying the one-way contract hash and verdict evidence — never the resolved selector values. |
docs/EFFECT_KIT.md |
doc | artifact (doc/benchmark) | roadmap | The operator-facing kit contract: config reference plus Frappe (REST/SQL) and OpenEMR (FHIR/SQL) worked examples targeting the reference fixtures. |
Caveats (honest limits):
- CONTRACT-PROVEN, not live-proven: the SQL verifier is exercised against sqlite fixtures (no production MariaDB/Postgres), the SFTP arrival path against a fake transport (no real SFTP server), and the worked-example configs are templates — only the FHIR substrate has an additional opt-in live-OpenEMR test. Per-verifier claims are exactly as strong as these tests.
On ambiguity the runtime halts instead of guessing, and an identity gate refuses a wrong-entity (wrong-patient) click — on ARMED steps only.
- Surfaces: README.md, docs
- Strongest evidence strength: supported (tier is
supported)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_identity.py |
test | ci (required PR gate (test)) | supported | Wrong-entity / near-name sibling / generic-band / param-in-band probes: the identity check refuses to verify a mismatched target and never arms on non-discriminative bands. |
tests/test_disambiguation.py |
test | ci (required PR gate (test)) | supported | Underdetermined targets surface as explicit disambiguation rather than a guessed click. |
tests/test_identity_fuzz.py |
test | ci (required PR gate (test)) | supported | Public property-based checks preserve the never-false-accept invariant for collapsible identifiers while requiring clean identifiers to remain verifiable. |
tests/test_identity_ladder.py |
test | ci (required PR gate (test)) | supported | The identity resolution ladder (structural/template/OCR) escalation. |
Caveats (honest limits):
- Identity verification covers ONLY armed steps. Real bundles arm a MINORITY of clicks (4 of 12 on a recent live OpenEMR bundle); an UNARMED click proceeds with no identity check whatsoever. Coverage is an auditable per-step metric, but disclosure does not close the gap.
The compiler emits a pixel identifier crop (anchor.identifier_crop) for identity-armed steps without structured identity — and for any step whose identifying region is marked at record time (--identifier) — so the pixel-compare identity tier arms on remote-display/pixel replays; every crop-less identity-applicable step records an explicit degrade reason, and lint surfaces per-bundle pixel-identity coverage.
- Surfaces: docs
- Strongest evidence strength: supported (tier is
supported)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_compile_identifier_crop.py |
test | ci (required PR gate (test)) | supported | Synthetic-fixture proof of the emission mechanism: a pixel-only recording compiles WITH a crop under templates/identifiers/ (sealed + manifest-hashed like every image crop); a structured recording writes NO identity pixels and records why; explicit --identifier markings (event rect / meta region) win; encrypted save seals the crop and the sealed crop still reaches the tier; the compiled crop drives a wrong-MRN MISMATCH through the real replayer ladder while pixel VERIFY stays hard-gated off (zero-false-accept preserved). |
tests/test_identifier_crop_lint.py |
test | ci (required PR gate (test)) | supported | Lint coverage surfacing: identity-armed steps with vs without crops (missing_identifier_crop findings carry the compiler's degrade reason; warn on band-only identity, info under structured identity). |
Caveats (honest limits):
- Mechanism proven on synthetic fixtures and the bundled demo app only — this does NOT re-qualify any desktop/remote-display substrate, and the pixel tier remains MISMATCH-or-ABSTAIN (it can add a safe halt on a wrong identifier, never authorize a match; PIXEL_VERIFY_ENABLED stays off pending a jitter-robust distance).
- Automatic emission requires an OCR-readable identity band at compile time; icon-only or unreadable rows still compile crop-less (with the recorded reason) and fall back to the OCR band tier at replay.
A halt is learnable: halt -> teach a correction -> promote a guarded branch, refusing underdetermined fixes.
- Surfaces: README.md, docs
- Strongest evidence strength: supported (tier is
supported)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_halt_learn_loop.py |
test | ci (required PR gate (test)) | supported | A surprise modal halts and emits a learnable trace; teaching promotes a guarded dismiss branch that replays without regression; an underdetermined correction is refused. |
tests/test_teach_cli.py |
test | ci (required PR gate (test)) | supported | The teach CLI resolves a halt via a correction spec or a recording, refuses an underdetermined fix, and errors on a non-halted run. |
Caveats (honest limits):
- Promotion is governed by a regression gate that blocks identity- weakening corrections; the loop refuses to guess when the fix is underdetermined.
A qualification client can add or update a typed finite business-policy decision without editing an internal manifest. The compiled workflow can then pause at that decision, accept one authorized and attributed choice, retain a signed durable receipt, revalidate the live application, and continue only through the certified successor branch.
- Surfaces: docs
- Strongest evidence strength: supported (tier is
supported)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_business_decision.py |
test | ci (required PR gate (test)) | supported | Required CI covers typed qualification authoring and replacement, qualification-revision and certification invalidation, ambiguous insertion refusal, the closed runtime decision contract, role and evidence checks, expiry, renewal, idempotency, crash recovery, signed branch authority, scoped outputs, fresh live-state revalidation, normal downstream action gates, and repair-bypass refusal. |
docs/BUSINESS_DECISIONS.md |
doc | artifact (doc/benchmark) | roadmap | The public engine contract separates a declared business choice from an operational halt and defines the Desktop, Cloud, and customer-local integration boundary. |
Caveats (honest limits):
- The engine validates a principal and its roles; it does not authenticate a self-asserted user. Desktop, Cloud, or a customer-local identity route must supply the authenticated principal.
- A human choice is control authority only. It cannot satisfy entity identity, a postcondition, or effect verification, and it cannot turn a screen statement or a human assertion into VERIFIED.
A qualification project can bind typed local facts, evidence hashes, reviewer provenance, and the exact decision contract to reviewed examples and counterfactuals. It can retain permanent human authority, record a reviewed automatic-rule candidate, or refuse certification until more evidence exists, without converting one historical choice into policy.
- Surfaces: docs
- Strongest evidence strength: supported (tier is
supported)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_judgment_cases.py |
test | ci (required PR gate (test)) | supported | Required CI covers closed fact schemas, exact workflow and decision bindings, local-only evidence references, conflict refusal, reciprocal contrast coverage for automatic-rule candidates, retained human authority, unresolved-evidence refusal, and the invariant that case evaluation never edits or synthesizes the executable rule. |
tests/test_qualification_project.py |
test | ci (required PR gate (test)) | supported | Required CI binds the reviewed local evidence bytes and review note into certification, reproduces that exact contract when a qualified workflow is saved, and refuses changed case or evidence digests. |
docs/BUSINESS_DECISIONS.md |
doc | artifact (doc/benchmark) | roadmap | The public contract separates qualification-time judgment capture from one live runtime answer and describes the scriptable authoring and judgment-case interfaces. |
Caveats (honest limits):
- An automatic-rule case names only a reviewed rule identifier and finite option. Flow does not infer or generate executable policy from the case; the rule must be authored and qualified through the normal program path.
- A retained human-node case preserves runtime human authority. A more-evidence-required case refuses certification instead of guessing.
A negotiated V2 attended task binds optional reviewed entity wording to the exact qualification, bundle, step, policy, and pause authority. An unqualified or unrecognized entity class stays local and the remote task uses the signed neutral record or item fallback.
- Surfaces: docs
- Strongest evidence strength: supported (tier is
supported)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_attended_actions.py |
test | ci (required PR gate (test)) | supported | Required CI covers the V1-compatible and V2 signed envelopes, provenance binding, remote-safe entity vocabulary, idempotent decisions, and fresh revalidation before resumed actuation. |
tests/test_qualification_project.py |
test | ci (required PR gate (test)) | supported | Required CI requires the current exact qualification contract before the V2 producer can emit its reviewed presentation metadata. |
docs/DECISION_DELIVERY.md |
doc | artifact (doc/benchmark) | roadmap | The delivery contract defines explicit V2 negotiation, neutral V1 fallback, and the local boundary for actual entity identifiers. |
Caveats (honest limits):
- V2 requires openadapt-types 0.10.x and explicit peer negotiation. The dependency does not upgrade a V1 consumer, and the byte-compatible V1 path remains available.
- The entity class is presentation metadata only. Actual record identity remains inside the customer-controlled runner and is revalidated before any resumed action.
The MockMed benchmark is CI-reproducible: compiled replay finishes ~4.9s p50 with zero model calls versus
37.5s p50 ($0.27/run) for the agent. Measured 2026-07-08 on Flow 0.1.0, a pre-v0.2.0 source build; not re-measured on a later release.
- Surfaces: README.md, docs
- Strongest evidence strength: supported (tier is
supported)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_mockmed.py |
test | ci (required PR gate (test)) | supported | The MockMed app + drift screens are deterministic and reachable, the substrate the benchmark replays against. |
tests/test_benchmark.py |
test | ci (required PR gate (test)) | supported | The benchmark harness / report accounting that produces the numbers. |
benchmark/BENCHMARK.md |
benchmark | artifact (doc/benchmark) | roadmap | Full numbers, methodology, and caveats for the CI-reproducible MockMed comparison (dated run artifact). |
Caveats (honest limits):
- The published latency/cost figures come from a dated run artifact measured on Flow 0.1.0 (pre-v0.2.0 source checkout; exact runtime HEAD not retained) on 2026-07-08; the app is intentionally simple, so success rate (100/100 vs 20/20) is not the story — the standing cost/latency delta is. Agent cost is a list-price estimate.
Windows UIA has scoped acceptance: the deterministic structural rung delivered native actions in a fixed WinForms workflow with independent effects and fail-closed stale/ambiguity refusal.
- Surfaces: README.md, docs, website
- Strongest evidence strength: supported (tier is
validating)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/e2e/test_parallels_desktop_e2e.py |
test | opt-in (OAFLOW_PARALLELS_E2E) | validating | OPT-IN live proof on a real Win11-ARM VM: record->compile->replay through WindowsBackend with the structural rung resolving every click by AutomationId (armed_coverage == 1.0). |
tests/test_windows_backend.py |
test | ci (required PR gate (test)) | supported | The 4-method Backend protocol on Windows against a MOCKED WAA server (runs in CI post-merge; proves the wire, not a live desktop). |
benchmark/structural_action/STRUCTURAL_ACTION.md |
benchmark | artifact (doc/benchmark) | roadmap | Desktop drift benchmark: the structural rung resolved 21/21 targets where visual replay alone managed 6/21. |
benchmark/windows_uia/results.json |
benchmark | artifact (doc/benchmark) | roadmap | Accepted candidate 20260717-candidate-56759c8-v2: 3/3 exact task and SQLite effects, 12 UIA receipts, and 3/3 stale plus 3/3 ambiguity refusals with zero silent incorrect success, over-halt, or model calls. |
Caveats (honest limits):
- The live desktop e2e is OPT-IN and infra-gated (needs a Parallels Win11-ARM VM;
OAFLOW_PARALLELS_E2E=1); it never runs on default CI. Only the mocked backend protocol runs in CI. - Acceptance covers the named in-tree workflow and exact Windows VM. Third-party applications require workflow-specific qualification.
Native macOS has scoped acceptance for exact-window-bound text delivery, independent file effects, and ambiguity refusal on a fixed TextEdit task.
- Reproducibility: field
- Surfaces: README.md, docs, website
- Strongest evidence strength: supported (tier is
validating)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_macos_backend.py |
test | ci (required PR gate (test)) | supported | CI covers unique-window binding, foreground/focus proof, exact-element text delivery, and refusal before physical input when proof fails. |
tests/test_macos_qualification.py |
test | ci (required PR gate (test)) | supported | CI validates the qualification oracle, failure taxonomy, evidence hashes, cleanup adjudication, and ambiguity-refusal accounting. |
benchmark/macos_native/textedit_counted_3plus1_b1b61a5_20260717.adjudication.json |
benchmark | artifact (doc/benchmark) | roadmap | Candidate b1b61a5 completed 3/3 exact-byte TextEdit effects and refused two ambiguous windows without modifying either file. |
Caveats (honest limits):
- Acceptance covers TextEdit on one macOS 15.7.3 Apple Silicon host and active user session; each customer application is qualified separately.
- The original counted report remains failed. The hash-bound adjudication accepts only its independently confirmed action/effect/refusal evidence.
Native Linux uses an exact-window AT-SPI backend with complete candidate enumeration, native action delivery, and fail-closed ambiguity, staleness, and Wayland portal boundaries.
- Surfaces: README.md, docs
- Strongest evidence strength: supported (tier is
validating)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_linux_backend.py |
test | ci (required PR gate (test)) | supported | Default CI covers exact app/window binding, bounded candidate enumeration, ambiguity and stale-target refusal, native invoke/focus and editable-text delivery, window-scoped capture, physical-input opt-in, and Wayland portal refusal through an injected Linux client. |
tests/test_linux_qualification.py |
test | ci (required PR gate (test)) | supported | Default CI enforces the live qualification's fixed three-trial matrix, independent exact-file and absence oracles, and separate silent-incorrect-success, over-halt, and refusal-failure accounting. |
scripts/qualify_linux_atspi.py |
benchmark | artifact (doc/benchmark) | roadmap | The required linux-atspi-x11 job drives the real GTK3 fixture through AT-SPI on an isolated Xvfb display and session D-Bus, emitting a per-trial JSON evidence artifact. |
docs/desktop/LINUX_NATIVE.md |
doc | artifact (doc/benchmark) | roadmap | The operator contract, X11 and Wayland boundaries, configuration, delivery-only receipt semantics, and required real GTK/AT-SPI CI qualification contract. |
Caveats (honest limits):
- Required CI establishes live scoped acceptance only for the in-tree GTK3 application on its isolated Xvfb image. Each third-party application and environment requires its own qualification.
- X11 is the initial built-in live transport. Wayland requires a real operator-approved XDG RemoteDesktop/ScreenCast portal session; the built-in client currently refuses rather than pretending a grant exists.
RDP has scoped acceptance for real-network framebuffer decode and input delivery into Windows 11 with independent exact file-effect verification.
- Reproducibility: field
- Surfaces: README.md, docs, website
- Strongest evidence strength: supported (tier is
validating)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/e2e/test_parallels_rdp_e2e.py |
test | opt-in (OAFLOW_PARALLELS_RDP_E2E) | validating | Opt-in snapshot-safe real RDP qualification harness with exactly three trials, independent guest-tools oracle, failure taxonomy, and cleanup. |
tests/test_rdp_backend.py |
test | ci (required PR gate (test)) | supported | CI covers the backend/transport contract, framebuffer conversion, pointer and keyboard delivery, resize rebaseline between actions, mid-lease geometry refusal, pre-delivery horizontal-scroll refusal, typed uncertain transport failures, and record-compile-replay conformance. |
tests/test_rdp_multiapp_campaign_contract.py |
test | ci (required PR gate (test)) | supported | Required CI covers the bounded 27-trial FreeRDP campaign contract, independent SQLite, CSV, and Maildir oracles, fail-closed visual action preflight, uncertain-delivery handling, and result accounting. |
benchmark/rdp_multiapp/README.md |
doc | artifact (doc/benchmark) | roadmap | The public synthetic multi-window campaign defines the workflow, faults, independent persisted surfaces, and acceptance denominator. |
benchmark/rdp/results_82a658a_20260718.sanitized.json |
benchmark | artifact (doc/benchmark) | roadmap | Candidate 82a658a passed 3/3 at 51.845, 10.467, and 7.477 seconds, with zero failures, silent incorrect successes, over-halts, or model calls. |
Caveats (honest limits):
- Acceptance covers the named 1280x800 Windows Run/file task and exact VM environment, not arbitrary applications, identity policies, or Citrix.
- The multi-window campaign uses deterministic synthetic applications. It exercises the real FreeRDP pixel and input path but does not qualify a named customer application or environment.
Runtime-validation v3 binds Standard and Regulated hosted activation to the exact governed authorization template reproduced from the sealed qualification. It also binds the successful run's resolved target kind (
web|windows|macos|linux|rdp|citrix), artifact, compiler, parameter schema, strict lint, policy, risk, report, and opaque environment. Web retains its exact HTTPS/host boundary. Native and remote execution emits no app, window, host, readiness, or backend-hint values to Cloud. Local verification retains signed v1 and v2 compatibility.
- Surfaces: README.md, docs, website
- Strongest evidence strength: supported (tier is
supported)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_runtime_validation.py |
test | ci (required PR gate (test)) | supported | Required CI exercises all six target kinds, validates v3 against its closed JSON Schema, refuses a Standard or Regulated activation without an exact template, binds the template SHA to the HMAC, preserves web URL/host checks, keeps native/remote execution empty, verifies the canonical HMAC vector, and retains signed v1/v2 compatibility. |
tests/test_cli_new_commands.py |
test | ci (required PR gate (test)) | supported | Required CI carries the resolved web, Windows, and Citrix backend token through durable resume into the new run report rather than trusting an upload-time relabel. |
schemas/runtime-validation-attestation-v3.json |
doc | artifact (doc/benchmark) | roadmap | The public closed envelope requires target_kind and the governed authorization template SHA, and conditionally permits browser execution fields only for web. |
Caveats (honest limits):
- The HMAC proves operator-token possession and envelope integrity; it is not independent observation of the local replay or certification of an arbitrary target application.
The product CLI records desktop workflows through openadapt-capture and
record --backend windows|macos|linux|rdp|citrixcaptures the operator's real demonstration and emits the compile-ready recording format, so record -> compile -> replay closes on the desktop substrate, not just the browser.
- Surfaces: README.md, docs, website
- Strongest evidence strength: supported (tier is
validating)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_desktop_record.py |
test | ci (required PR gate (test)) | supported | The live-capture orchestration + CLI wiring (record --backend windows |
tests/test_capture_adapter.py |
test | ci (required PR gate (test)) | supported | The capture->recording bridge over a REAL openadapt-capture session (its own event-processing pipeline + frame extraction) feeding the UNMODIFIED compiler (runs on default CI: the test job installs the capture extra; openadapt-capture >=0.5.4 imports clean headless). |
docs/desktop/RECORDING.md |
doc | artifact (doc/benchmark) | roadmap | The capture-assisted and live-observer paths, fail-closed secret handling, RDP coordinate binding, and reuse of openadapt-capture + the capture adapter. |
Caveats (honest limits):
- Offline capture carries NO structural (UIA AutomationId) locator, so replay uses the visual ladder (template/ocr/geometry). The live capture needs a display (openadapt-capture is the optional
captureextra); the CI proof covers a real on-disk capture session -> convert -> compile -> replay, not a live recorder run (the live recorder has its own integration suite in the openadapt-capture repo). --secretis refused on an unqualified pixel/desktop capture with no field geometry, and RDP recording uses the replay pixel space or a deployment-calibrated client-window mapping.
Citrix uses a dedicated exact-Workspace-window backend beneath the governed pixel-only remote-display contract. Required CI covers its readiness gate, governed execution, durable resume, report binding, and refusal contracts; the counted no-DOM qualification covers the driver, visual ladder, independent effects, and drift halts. Live ICA/HDX acceptance remains separately bound to the exact deployment.
- Surfaces: README.md, docs, website
- Strongest evidence strength: supported (tier is
validating)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/e2e/test_citrix_pixel_e2e.py |
test | opt-in (OAFLOW_CITRIX_PIXEL_E2E) | validating | OPT-IN pixel-only proof: structural_armed_coverage == 0, replay resolves on template/ocr/geometry only, on-screen OCR read-back verifies the write, the identity gate HALTs on a look-alike patient, and render drift triggers halt-on-ambiguity. |
tests/test_pixel_identity_probe.py |
test | ci (required PR gate (test)) | supported | The pixel-substrate identity probe behind the on-pixels identity gate. |
tests/test_citrix_workspace_backend.py |
test | ci (required PR gate (test)) | supported | Required CI covers the dedicated backend preset, exact owner/title overrides, readiness probe construction, pixel-only capability boundary, backend factory, and product CLI selectors. |
tests/test_run_gate.py |
test | ci (required PR gate (test)) | supported | Required CI refuses governed Citrix execution before action when its readiness binding is absent or blank, admits a sealed bundle carrying a recorded readiness binding, and reports only the resolved backend token rather than sensitive target strings. |
tests/test_cli_new_commands.py |
test | ci (required PR gate (test)) | supported | Required CI restores the recorded Citrix owner, exact title, and readiness binding through durable approve/resume, and refuses before backend construction when resumed configuration is incomplete. |
tests/test_hosted.py |
test | ci (required PR gate (test)) | supported | Required CI binds a completed local Citrix report to the closed citrix execution token without copying target owner, title, or readiness values into the hosted summary. |
tests/e2e/test_citrix_workspace_standin_e2e.py |
test | opt-in (OAFLOW_CITRIX_STANDIN_E2E) | validating | The dedicated Citrix backend passes three healthy effect-confirmed record->compile->replay trials and three severe-drift safe-halts over a no-DOM surface, with zero model calls, false completion, or silent incorrect success. |
tests/test_citrix_real_acceptance.py |
test | ci (required PR gate (test)) | supported | Required CI validates the public real-ICA campaign preflight, distinct authority keys, executable and oracle attestations, one-use nonce journal, crash recovery, signed display and monitor-topology observation, explicit reliability metrics, uncertain dispatch, and fail-closed report contract without provisioning infrastructure. |
benchmark/citrix_workspace/results.json |
artifact | artifact (doc/benchmark) | roadmap | The retained six-trial record reports code_readiness_accepted=true and ica_hdx_accepted=false, keeping driver readiness distinct from a counted live ICA/HDX qualification. |
docs/desktop/CITRIX_PIXEL.md |
doc | artifact (doc/benchmark) | roadmap | The Citrix driver model, reusable evidence, exact-deployment acceptance contract, independent-effect boundary, and customer-controlled posture. |
Caveats (honest limits):
- PIXEL-ONLY: no UIA/DOM crosses to the driver. On-screen OCR read-back is SAME-SURFACE, not an independent system-of-record check.
- A Parallels VM window is a Citrix ANALOG; real HDX/latency/DPI/lock- screen drift is not simulated. OPT-IN + infra-gated (
OAFLOW_CITRIX_PIXEL_E2E=1orOAFLOW_CITRIX_STANDIN_E2E=1); neither live-window proof runs on default CI, and neither replaces a counted live ICA/HDX acceptance record. - The public real-acceptance harness contains no customer recipe, credential, identifier, screenshot, or deployment-specific threshold. A private qualified configuration supplies those exact bindings.
Windows hosts can replay window-scoped remote-display workflows (the Citrix/RDP client window on a Windows endpoint) through the same governed pixel contract already proven on macOS hosts.
- Surfaces: docs
- Strongest evidence strength: supported (tier is
roadmap)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_win32_window_client.py |
test | ci (required PR gate (test)) | supported | MOCK-ONLY conformance: the win32 WindowClient implements the remote-display WindowClient seam (exact process/title/class selection with ambiguity halt, client-area coordinate mapping under per-monitor DPI awareness, PrintWindow-then-BitBlt capture fallback, focus-verification refusal, and the UIPI elevation guard) against a scripted Win32 API — never against a real Windows host. |
docs/desktop/CITRIX_PIXEL.md |
doc | artifact (doc/benchmark) | roadmap | The remote-display contract the client implements and the evidence ladder for qualifying it (Parallels/RDP client-window analog first, then a counted live Citrix host run). |
Caveats (honest limits):
- NOT qualified: no counted end-to-end batch on a real Windows host or a real client window exists. The contract tests mock every Win32 call; real DPI virtualization, UIPI, foreground-arbitration, and PrintWindow rendering behavior remain unproven until the Parallels/real-Windows validation procedure is executed and counted.
On the real third-party OpenEMR public demo, compiled replay went 19/20 versus 10/10 for a computer-use agent, faster and with zero model calls. Measured 2026-07-08 on Flow 0.1.0, a pre-v0.2.0 source build.
- Reproducibility: field
- Surfaces: README.md, docs, website
- Strongest evidence strength: supported (tier is
validating)
| Backing evidence | Kind | Gating / CI stage | Strength | Proves |
|---|---|---|---|---|
tests/test_openemr_benchmark.py |
test | ci (required PR gate (test)) | supported | The CI-reproducible half: the saved-message-row verifier, its unsaved-entry-form rejection, and the intent-not-coordinates task prompt that both benchmark arms use. |
tests/test_openemr_retained_finals.py |
test | opt-in (OPENADAPT_OPENEMR_FINALS_DIR) | validating | When the local-only retained final frames are mounted, the guard replays all 30 with the current verifier and requires 19/20 compiled, 10/10 agent, and exactly one corrected legacy false success. |
benchmark/openemr/BENCHMARK.md |
benchmark | artifact (doc/benchmark) | roadmap | The corrected field-test numbers, saved-row screen-oracle contract, methodology, and cost caps for the 20-vs-10 head-to-head run. |
docs/showcase-openemr/FINDINGS.md |
doc | artifact (doc/benchmark) | roadmap | The correctness-only field findings (fresh browsers, zero model calls, closed-loop scrolling). |
Caveats (honest limits):
- The historical result uses OCR evidence from a visible saved message row. It does not use an out-of-band OpenEMR system-of-record read.
- FIELD TEST, NOT CI-reproducible: the head-to-head ran against a SHARED public demo that other users mutate and that resets daily; the sample is small (10 agent runs). Only the verifier + task-prompt units run in CI.
reproducibility: fieldforbids ever labeling thissupported.