diff --git a/AGENTS.md b/AGENTS.md index e51f8ce33..09b433bea 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -203,10 +203,10 @@ This repo encodes invariants as self-declaring gates. The correct response to a ## Selector System Rules - Interaction commands (`click`, `fill`, `get`, `is`) and `wait` accept selectors and `@ref`. -- Pipeline: **parse -> resolve -> act -> record selectorChain -> heal on replay**. +- Pipeline: **parse -> resolve -> act -> record selectorChain -> re-resolve as a divergence suggestion on replay failure**. - Keep selector parsing, matching, and resolution in `src/selectors/`. - Call `buildSelectorChainForNode` after resolving target nodes. -- New element-targeting interactions must support selector + `@ref`, record `selectorChain`, and hook replay healing (`healReplayAction` in `session.ts` + selector helpers in `session-replay-heal.ts`). +- New element-targeting interactions must support selector + `@ref` and record `selectorChain`, so `collectReplaySelectorCandidates` (`src/daemon/handlers/session-replay-heal.ts`) can surface it as a ranked suggestion in a replay divergence report (`session-replay-divergence.ts`). ADR 0012 retired `--update`'s silent rewrite-on-heal; there is no automated write path to hook into. - New selector keys remain centralized in `src/selectors/parse.ts`. - New `is` predicates belong in `evaluateIsPredicate`. - On macOS, snapshot rects are absolute in window space. Point-based runner interactions must translate through the interaction root frame; do not assume app-origin `(0,0)` coordinates. diff --git a/docs/adr/0012-interactive-replay.md b/docs/adr/0012-interactive-replay.md index 24df42052..4ad3752ba 100644 --- a/docs/adr/0012-interactive-replay.md +++ b/docs/adr/0012-interactive-replay.md @@ -441,10 +441,18 @@ indexing, so repeated source lines are distinguished by their plan index. Every divergence includes `resume: { allowed, from, reason?, planDigest }`. `planDigest` is SHA-256 over the canonical fully expanded plan, including each action's command, normalized inputs, control shape, -platform-conditioned expansion, and source provenance. A resume requires both `--from N` and +platform-conditioned expansion, and source provenance. Concretely "normalized inputs" bind each action's +positionals/flags, its execution-affecting `runtime` hints, and its `target-v1` identity annotation +(decision 3 — verification consumes it pre-action, so a changed annotation is execution-affecting); and +"platform-conditioned expansion" binds the EFFECTIVE resolved `platform`/`target` the run invokes with (CLI +flag over script metadata), never the raw script metadata, so a digest computed for one target is not +reusable against another. Deliberately EXCLUDED are native `.ad` `${VAR}` VALUES: substitution happens +after planning, so changing only late-bound values keeps the same digest. Maestro environment substitution +instead happens during compatibility parsing and can change action inputs, includes, or control expansion, +so it can change the digest. A resume requires both `--from N` and `--plan-digest ` from the report. The daemon rebuilds the current plan and rejects -`INVALID_ARGS` before any action when its digest differs, so edits, include changes, or environment-driven -expansion cannot silently retarget ordinal N. `allowed: false` explains why no resume is safe; its digest +`INVALID_ARGS` before any action when its digest differs, so edits or parse-time expansion cannot silently +retarget ordinal N. `allowed: false` explains why no resume is safe; its digest is still diagnostic, not an authorization to bypass preflight. Resume does not reconstruct execution state. For `N > 1`, preflight must reject with `INVALID_ARGS` when diff --git a/docs/maestro-compat-debt-map.md b/docs/maestro-compat-debt-map.md index 44be53118..d063cef97 100644 --- a/docs/maestro-compat-debt-map.md +++ b/docs/maestro-compat-debt-map.md @@ -9,7 +9,7 @@ This map summarizes the Maestro compatibility surface after the lane 6 audit. LO | Input and focus | `replay-flow.ts` input coalescing, `interactions.ts` input/erase/paste/pressKey slices, `runtime.ts`, daemon Maestro fallback in `interaction-touch.ts` | ~350 LOC plus shared daemon flags | Coalesces `tapOn` + `inputText` + `pressKey Enter` into `wait`/`fill`/enter for likely text-entry selectors, leaves focused-field input as `type`, maps `eraseText` to backspaces, falls back from keyboard enter to newline typing, and allows Maestro non-hittable coordinate fallback for tap selectors. | Uses native `fill`, `type`, `keyboard`, `click`, replay variable substitution, and daemon touch fallback flags. | A native "focus then fill" replay primitive and focused-field clear/erase operation would remove most parser heuristics while improving `.ad` flows too. | Text-entry detection is name-based; focused-field commands depend on existing device focus; non-hittable coordinate fallback is intentionally compatibility-only and can mask poor selectors. | Moderate: parser coalescing tests in `replay-flow.test.ts`; daemon fallback covered in `interaction.test.ts`; no broad provider guard for focused erase/paste. | Converge on native focus/fill/clear semantics. Keep Maestro's optional/non-hittable quirks as compat flags. | Lane 8: native input primitive | | Assertions and waits | `runtime-assertions.ts`, wait/assert slices of `interactions.ts`, `runtime-flow.ts` visible condition path | ~450 LOC | Polls raw snapshots for `assertVisible`, adds a terminal grace capture, treats `assertNotVisible` as stable hidden after repeated samples or timeout, computes animation stability from snapshot signatures, maps `extendedWaitUntil`, and waits briefly for `runFlow.when.visible` while keeping `notVisible` point-in-time. | Uses native `snapshot`, `wait`, `is`, visible predicates, reference frames, replay blocks, and timeout helpers. | A shared waiter/poller utility with explicit grace, stable-hidden, raw-snapshot, and timeout policies would let native `wait` and Maestro share mechanics without sharing defaults. | Timeout and stability semantics are subtle; raw full snapshots are expensive; making `notVisible` wait would change cleanup-flow behavior. | Good: `runtime-assertions.test.ts` covers deadline/hidden edge cases; `runtime-flow.test.ts` covers visible wait and immediate notVisible. | Extract generic polling/stability helpers; keep Maestro timing constants and condition semantics local. | Lane 8: waiter convergence | | Scroll, swipe, and geometry | `points.ts`, `runtime-geometry.ts`, geometry/scroll/swipe slices of `interactions.ts` and `runtime-interactions.ts` | ~500 LOC | Parses absolute and percentage points, converts authored percentage swipes through the shared core frame-bounds helper, caches reference frames in replay scope, delegates directional screen swipes to core gesture presets, biases tap points for large text containers and bottom tabs, loops `scrollUntilVisible` with `wait`/`find` probes, and maps `swipe.label`. | Uses native `click`, `gesture`, `scroll`, `swipe`, `find`, `wait`, touch reference frames, and raw snapshots. | Target-derived swipes and frame caching could move into native gesture planning when non-Maestro callers need them. | Stale or missing raw snapshot frames break percent gestures; target-derived swipe geometry and tap bias remain compatibility heuristics. | Good: `runtime-geometry.test.ts`, `runtime-interactions.test.ts`; the provider guard asserts fresh snapshots and exact authored Android percentage coordinates. | Keep authored coordinates lossless and route semantic gestures through core planners. Extract target geometry or frame caching only with a second native caller. | Lane 8 or 9: gesture planner | -| Flow control, `runFlow`, retry, and hooks | `flow-control.ts`, `runtime-flow.ts`, hook flattening in `replay-flow.ts` | ~700 LOC | Handles `onFlowStart`/`onFlowComplete`, file and inline `runFlow`, per-block env, static platform gates, limited `when.true` boolean/platform expressions, runtime visible/notVisible gates through `SessionAction.replayControl`, parse-expanded `repeat.times`, and runtime `retry` via replay retry blocks. | Depends on replay block runtime (`invokeReplayActionBlock`, `invokeReplayRetryBlock`), replay vars/env, typed `SessionReplayControl`, and native snapshot visibility resolution. | Flow block execution is now mostly converged. A native replay block AST would only be worth it if native `.ad` syntax also needs conditional blocks, deterministic repeats, or faithful nested line reporting. | `repeat.times` materializes actions with a guardrail; nested line numbers are lossy; expression support is intentionally tiny; visible conditions are snapshot-dependent; compat flow controls are intentionally in-memory and rejected for `replay -u`. | Strong: `replay-flow.test.ts` covers hooks/runFlow/env/repeat/retry/platform gates/expressions; `runtime-flow.test.ts` covers runtime condition behavior; daemon replay tests guard execution and `replay -u` rejection; PR #620 covers provider retry/runFlow path. | Keep `replayControl` as the shared runtime boundary. Do not add a native block AST unless native replay gains its own block syntax; keep Maestro expression grammar and unsupported `repeat.while` local until native runtime has loop semantics. | Lane 7 complete / revisit only with native block syntax | +| Flow control, `runFlow`, retry, and hooks | `flow-control.ts`, `runtime-flow.ts`, hook flattening in `replay-flow.ts` | ~700 LOC | Handles `onFlowStart`/`onFlowComplete`, file and inline `runFlow`, per-block env, static platform gates, limited `when.true` boolean/platform expressions, runtime visible/notVisible gates through `SessionAction.replayControl`, parse-expanded `repeat.times`, and runtime `retry` via replay retry blocks. | Depends on replay block runtime (`invokeReplayActionBlock`, `invokeReplayRetryBlock`), replay vars/env, typed `SessionReplayControl`, and native snapshot visibility resolution. | Flow block execution is now mostly converged. A native replay block AST would only be worth it if native `.ad` syntax also needs conditional blocks, deterministic repeats, or faithful nested line reporting. | `repeat.times` materializes actions with a guardrail; nested line numbers are lossy; expression support is intentionally tiny; visible conditions are snapshot-dependent; compat flow controls are intentionally in-memory (a `retry`/`runFlow.when` block is one entry in the replay-resume executable plan, never individually addressable by `replay --from`, and ADR 0012 migration step 5's resume preflight rejects skipping into or resuming at one). | Strong: `replay-flow.test.ts` covers hooks/runFlow/env/repeat/retry/platform gates/expressions; `runtime-flow.test.ts` covers runtime condition behavior; daemon replay tests guard execution and resume's control-flow rejection; PR #620 covers provider retry/runFlow path. | Keep `replayControl` as the shared runtime boundary. Do not add a native block AST unless native replay gains its own block syntax; keep Maestro expression grammar and unsupported `repeat.while` local until native runtime has loop semantics. | Lane 7 complete / revisit only with native block syntax | | `runScript` | `run-script.ts`, `runtime.ts` runScript branch | 229 LOC plus router branch | Executes trusted flow-local JavaScript with `node:vm`, exposes env values, `output`, `json`, and synchronous-looking `http.post` through a timeout-bounded child Node process; exports output as `output.` replay variables. | Uses replay variable scope, `runCmdSync`, and `AppError`; there is no native `.ad` command equivalent. | Shared env/output variable plumbing could converge, but script execution should not become native without a separate security model and product decision. | High security and determinism risk: `node:vm` is not a sandbox, scripts can make network requests, async support is intentionally narrow, and output key rules are compatibility-specific. | Partial: parser/order/env/path behavior covered in `replay-flow.test.ts`; docs describe trust/security limits; no focused execution tests for `http.post`, `json`, or output validation. | Keep compat-local. Add focused execution tests before expanding helpers; do not expose as native command until sandboxing and trust model are explicit. | Lane 9: runScript guard tests | | Suite discovery and test artifacts | `src/daemon/handlers/session-test-discovery.ts`, `session-test-suite.ts`, `session-test-artifacts.ts`, replay grammar/backend plumbing | ~120 Maestro-specific LOC in daemon/test path | When `--maestro`/backend is selected, discovers `.ad`, `.yaml`, and `.yml`, allows untyped YAML through platform filtering, runs through native replay test suite, and preserves original Maestro flow filenames in artifacts. | Native test suite runner, replay backend selection, session lifecycle, artifact materialization, and CLI `--maestro` flag. | Mostly converged already. The remaining improvement is making replay backend extension/filter policy data-driven so future backends avoid daemon conditionals. | Low. Main risk is accidentally including non-Maestro YAML when backend is set, or changing platform-filter behavior for untyped YAML. | Good: `session-test-discovery.test.ts`, `session-test-suite.test.ts`, `session-test-artifacts.test.ts`; PR #620 provider suite runs a Maestro YAML test. | Keep small daemon hook for now; extract backend discovery policy only if another replay backend appears. | Lane 6 complete / no follow-up unless backend count grows | | Docs and support matrix | `support-matrix.ts`, `website/docs/docs/replay-e2e.md`, CLI flag/help tests, issue tracker references | 42 LOC source matrix plus docs | Maintains supported/unsupported capability lists, formats CLI help copy, links tracker/new issue URLs, and keeps replay docs synced with the source matrix. | CLI flag definitions/help rendering and website docs. | Generate or embed the support list from one source in docs/help to remove manual prose drift. | Medium user-facing risk: stale docs can imply unsupported parity or hide known gaps. | Good: `support-matrix.test.ts` asserts CLI help and docs stay synced with the shared matrix. | Keep `support-matrix.ts` as source of truth; update it with every behavior change and mirror only explanatory context in docs. | Lane 6 complete / ongoing docs hygiene | diff --git a/scripts/integration-progress-model.ts b/scripts/integration-progress-model.ts index f98f96ee2..49d79cfd1 100644 --- a/scripts/integration-progress-model.ts +++ b/scripts/integration-progress-model.ts @@ -174,8 +174,10 @@ function summarizeProviderScenarioFlagCoverage(files) { ['restart', 'logs clear --restart workflow'], ['networkInclude', 'network dump include modes', ['include']], ['noRecord', 'action recording suppression'], - ['replayUpdate', 'selector-healing replay update', ['update']], + ['replayUpdate', 'retired --update no-op replays without rewriting (ADR 0012)', ['update']], ['replayEnv', 'replay/test variable injection', ['env']], + ['replayFrom', 'replay resume skips completed steps (ADR 0012)', ['resumeFrom']], + ['replayPlanDigest', 'replay resume plan-digest preflight binding', ['resumePlanDigest']], ['failFast', 'test suite stops after first failure'], ['timeoutMs', 'wait/test timeout flags'], ['retries', 'test suite retry budget flows through request path'], diff --git a/src/__tests__/cli-help.test.ts b/src/__tests__/cli-help.test.ts index b32f5b063..3ce5bce72 100644 --- a/src/__tests__/cli-help.test.ts +++ b/src/__tests__/cli-help.test.ts @@ -76,6 +76,11 @@ test('help workflow prints agent workflow topic and skips daemon dispatch', asyn assert.match(result.stdout, /agent-device help workflow/); assert.match(result.stdout, /Core loop:/); assert.match(result.stdout, /Do not use CSS selectors/); + assert.match(result.stdout, /Native \.ad interpolation is late-bound after planning/); + assert.match( + result.stdout, + /Maestro environment substitution occurs during compatibility parsing/, + ); }); test('help workflow preserves known device workaround guidance', async () => { @@ -114,7 +119,10 @@ test('help workflow documents the selector disambiguation policy (#1037)', async assert.match(result.stdout, /deepest node first/); assert.match(result.stdout, /then smallest on-screen area/); assert.match(result.stdout, /Selector did not resolve uniquely/); - assert.match(result.stdout, /replay and replay-heal apply the same depth-then-area policy/); + assert.match( + result.stdout, + /replay's suggestion re-resolution.*applies the same depth-then-area policy/, + ); assert.match(result.stdout, /targetHittable: false/); }); diff --git a/src/cli/parser/cli-help.ts b/src/cli/parser/cli-help.ts index 3e19f00ec..e1e4f38d1 100644 --- a/src/cli/parser/cli-help.ts +++ b/src/cli/parser/cli-help.ts @@ -241,7 +241,7 @@ Selectors: agent-device press 'id="submit-order"' agent-device is visible 'label="Online"' agent-device get text 'id="quantity-value"' - Ambiguous selector disambiguation: a selector on an interactive command (press/click/fill/focus/longpress/scroll/swipe/pinch) that matches multiple elements does not fail by default. It auto-resolves deepest node first (largest depth in the tree), then smallest on-screen area; only an exact tie on both depth and area fails with "Selector did not resolve uniquely". replay and replay-heal apply the same depth-then-area policy for touch/fill/get-text, so recorded flows and live commands pick the same candidate. This exists because short/reused labels (tab + header + button with the same text, or a duplicated list-row label) are common in real apps; add id="..." or a longer/more specific text to force a different match instead of assuming ambiguous selectors always fail. + Ambiguous selector disambiguation: a selector on an interactive command (press/click/fill/focus/longpress/scroll/swipe/pinch) that matches multiple elements does not fail by default. It auto-resolves deepest node first (largest depth in the tree), then smallest on-screen area; only an exact tie on both depth and area fails with "Selector did not resolve uniquely". replay's suggestion re-resolution (in a divergence report) applies the same depth-then-area policy for touch/fill/get-text, so recorded flows and live commands pick the same candidate. This exists because short/reused labels (tab + header + button with the same text, or a duplicated list-row label) are common in real apps; add id="..." or a longer/more specific text to force a different match instead of assuming ambiguous selectors always fail. Selector match can still land on a non-interactive node (for example an off-screen map annotation that exact-matches text= while the real control has a longer label). Success responses for press/fill/click/ref targets carry targetHittable: false and a hint when the resolved element reports hittable: false, since the tap may have had no visible effect; treat that as a signal to verify with a snapshot or re-target by @ref/longer text, not as a command failure. Text entry: @@ -320,7 +320,7 @@ Validation and evidence: agent-device screenshot agent-device press 124 817 agent-device snapshot -i - Startup/CPU/memory/frame first pass: perf metrics --json (bare perf and metrics are aliases). Focused frame/jank health: perf frames --json. Memory-only sample: perf memory sample --json returns compact JSON with bounded top offenders. Heap/memgraph artifact escalation: perf memory snapshot --out heap.artifact; use --kind android-hprof on Android or --kind memgraph on supported Apple simulator/macOS app sessions. Android native profiling: perf cpu profile start|stop|report --kind simpleperf --out ; Android native traces: perf trace start|stop --kind perfetto --out . Artifact collectors return compact state/path/size metadata only; raw heap/profile/trace files stay on disk. Treat native perf output as the agent evidence: for example, a Perfetto stop can return state=stopped, outPath=/tmp/app.perfetto-trace, sizeBytes=5392410, and method=adb-shell-perfetto while the 5.3 MB raw trace stays in the artifact. This is better than raw dumps for agents because it is stable, bounded, and keeps large artifacts out of context. heapprofd is deferred until Perfetto plumbing is available. Replay maintenance: replay -u ./flow.ad. + Startup/CPU/memory/frame first pass: perf metrics --json (bare perf and metrics are aliases). Focused frame/jank health: perf frames --json. Memory-only sample: perf memory sample --json returns compact JSON with bounded top offenders. Heap/memgraph artifact escalation: perf memory snapshot --out heap.artifact; use --kind android-hprof on Android or --kind memgraph on supported Apple simulator/macOS app sessions. Android native profiling: perf cpu profile start|stop|report --kind simpleperf --out ; Android native traces: perf trace start|stop --kind perfetto --out . Artifact collectors return compact state/path/size metadata only; raw heap/profile/trace files stay on disk. Treat native perf output as the agent evidence: for example, a Perfetto stop can return state=stopped, outPath=/tmp/app.perfetto-trace, sizeBytes=5392410, and method=adb-shell-perfetto while the 5.3 MB raw trace stays in the artifact. This is better than raw dumps for agents because it is stable, bounded, and keeps large artifacts out of context. heapprofd is deferred until Perfetto plumbing is available. Replay divergence and resume: a failing replay/test step returns REPLAY_DIVERGENCE with a bounded report (screen digest, ranked selector suggestions, resume). Repair app state, then resume with replay --from --plan-digest (both from the report's resume field) to continue from the failed step without re-running earlier ones; resume never re-executes skipped steps, so app state is the caller's responsibility, and it is rejected with INVALID_ARGS when the plan digest is stale, --from is out of range, or the skipped range/target touches runtime control flow. The plan digest binds the script, its includes, the effective --platform/--target, and per-action runtime/identity. Native .ad interpolation is late-bound after planning, so changing only its values keeps the digest; Maestro environment substitution occurs during compatibility parsing and can change action inputs, includes, or control expansion, so it can change the digest. --from is replay-only; test rejects it. --update/-u no longer rewrites the script (ADR 0012) — it is a no-op kept for compatibility; every divergence already carries the same ranked suggestions. Recording: record start/stop. The default scope is app and expects an active session created by open ; this keeps app proof videos tied to the intended app session. Use record start --scope device/system to explicitly request whole-screen capture where the selected backend supports it, such as recordings that intentionally span multiple apps, home screen, settings, or app transitions. Use --max-size to cap the longest edge and --quality medium|high to choose output quality across Android and Apple targets. By default, stop burns touch overlays into the video; use record start --hide-touches for the fastest raw recording. Android record start publishes a durable device manifest. Android adb screenrecord has a 180s platform limit, so longer Android recordings are returned as multiple MP4 chunks while the daemon stays alive; after daemon restart, record stop recovers only manifest-owned chunks and warns when gesture overlays are unavailable. For gesture-heavy iOS simulator proof videos, prefer --hide-touches because overlay timing depends on a stable runner session while gestures are executing. Tracing: trace start ./trace.log, trace stop ./trace.log. Paths are positional. Stable known flow: batch ./steps.json, not workflow batch. Inline batch JSON example: @@ -382,7 +382,7 @@ React Native dev loop: Guarantees: Statements of fact for agents to reason from without probing them via trial commands. Each is backed by source in the agent-device repo; behavior changes land with an updated statement here. - Selector ambiguity: a selector on an interactive command that matches multiple elements does not fail by default. Resolution auto-disambiguates deepest node first, then smallest on-screen area; only an exact tie on both fails with "Selector did not resolve uniquely (...)". replay and replay-heal apply the same depth-then-area policy, so recorded and live commands pick the same candidate. + Selector ambiguity: a selector on an interactive command that matches multiple elements does not fail by default. Resolution auto-disambiguates deepest node first, then smallest on-screen area; only an exact tie on both fails with "Selector did not resolve uniquely (...)". replay's suggestion re-resolution applies the same depth-then-area policy, so recorded and live commands pick the same candidate. Hittability: iOS AX hittable:false on a resolved node does not block resolution or fail the command; non-hittable resolution is allowed by design because iOS AX hittable flags are unreliable on deep React Native trees. press/fill/click success responses carry targetHittable: false plus a hint when the resolved ref or selector target reports hittable: false, so treat that as a signal to verify with a snapshot or re-target, not as a failure. Open: on iOS, open without --relaunch dispatches a plain simctl launch, which is idempotent-foreground for an already-running app (it brings the process forward; it does not restart it). open --relaunch restarts the app; on iOS simulators (not real devices or macOS) this collapses to one simctl launch --terminate-running-process call instead of a separate terminate-then-launch, so relaunch is a single step there. Close and runner retention: close keeps a healthy iOS simulator XCTest runner warm by default so the next open on that device skips the runner build, unless --shutdown was requested, the session was recording, the session held a device lease, or the device used a scoped (non-default) simulator set — any of those tear the runner down on close. A retained runner auto-stops after an idle window (default 5 minutes) to release the device's runner lease for other daemons; set AGENT_DEVICE_IOS_RUNNER_IDLE_STOP_MS to override the window, or 0 to disable idle stop and retain until daemon exit. diff --git a/src/client/client-types.ts b/src/client/client-types.ts index 16563c576..58d58bb3c 100644 --- a/src/client/client-types.ts +++ b/src/client/client-types.ts @@ -817,12 +817,26 @@ export type FindOptions = export type ReplayRunOptions = AgentDeviceRequestOverrides & AgentDeviceSelectionOptions & { path: string; + /** + * @deprecated ADR 0012 migration step 6: `--update` no longer rewrites + * the script. Accepted for backward compatibility; every divergence + * already carries ranked selector suggestions regardless of this flag. + */ update?: boolean; /** @deprecated Use backend: 'maestro'. */ maestro?: boolean; backend?: string; env?: string[]; timeoutMs?: number; + /** + * ADR 0012 decision 4 / migration step 5: resume at this 1-based plan + * step, skipping `1..resumeFrom-1` without executing them. Requires + * `resumePlanDigest` from the divergence report that reported this + * step as the failure. `replay` only — `test` has no resume fields. + */ + resumeFrom?: number; + /** The `resume.planDigest` from the divergence report `resumeFrom` came from. */ + resumePlanDigest?: string; }; export type ReplayTestOptions = AgentDeviceRequestOverrides & @@ -1000,6 +1014,8 @@ type CommandExecutionOptions = Partial & { replayBackend?: string; replayEnv?: string[]; replayShellEnv?: Record; + replayFrom?: number; + replayPlanDigest?: string; failFast?: boolean; timeoutMs?: number; retries?: number; diff --git a/src/commands/cli-grammar/flag-definitions-workflow.ts b/src/commands/cli-grammar/flag-definitions-workflow.ts index 10b0388a7..4992319d7 100644 --- a/src/commands/cli-grammar/flag-definitions-workflow.ts +++ b/src/commands/cli-grammar/flag-definitions-workflow.ts @@ -11,7 +11,28 @@ export const WORKFLOW_FLAG_DEFINITIONS: readonly FlagDefinition[] = [ names: ['--update', '-u'], type: 'boolean', usageLabel: '--update, -u', - usageDescription: 'Replay: update selectors and rewrite replay file in place', + usageDescription: + 'Replay: retired as a rewrite (ADR 0012) — never edits the .ad file; every divergence already ' + + 'carries ranked selector suggestions, --update or not', + }, + { + key: 'replayFrom', + names: ['--from'], + type: 'int', + min: 1, + usageLabel: '--from ', + usageDescription: + 'Replay: resume at 1-based plan step n, skipping 1..n-1 without executing them (requires ' + + "--plan-digest; see a divergence report's resume field). replay only, not test", + }, + { + key: 'replayPlanDigest', + names: ['--plan-digest'], + type: 'string', + usageLabel: '--plan-digest ', + usageDescription: + 'Replay: the plan digest a --from resume must match (from a prior divergence report); mismatch, ' + + 'edits, or include/platform-expansion changes fail INVALID_ARGS before any action', }, { key: 'replayMaestro', diff --git a/src/commands/cli-grammar/types.ts b/src/commands/cli-grammar/types.ts index b87eb44d7..f17944729 100644 --- a/src/commands/cli-grammar/types.ts +++ b/src/commands/cli-grammar/types.ts @@ -54,6 +54,11 @@ export type CommandInput = Omit query?: string; retainPaths?: boolean; retentionMs?: number; + // ADR 0012 decision 4 / migration step 5: replay-only resume. Named + // `resumeFrom`/`resumePlanDigest` (not `from`/`planDigest`) — `from` is + // already a gesture `PointInput` on this shared flat type. + resumeFrom?: number; + resumePlanDigest?: string; scale?: number; selector?: string; source?: InternalRequestOptions['installSource']; diff --git a/src/commands/command-flags.ts b/src/commands/command-flags.ts index 7aba06e40..9cb784364 100644 --- a/src/commands/command-flags.ts +++ b/src/commands/command-flags.ts @@ -91,6 +91,8 @@ function buildFlags(options: InternalRequestOptions): CommandFlags { replayBackend: options.replayBackend, replayEnv: options.replayEnv, replayShellEnv: options.replayShellEnv, + replayFrom: options.replayFrom, + replayPlanDigest: options.replayPlanDigest, failFast: options.failFast, timeoutMs: options.timeoutMs, retries: options.retries, diff --git a/src/commands/replay/index.test.ts b/src/commands/replay/index.test.ts index 5e619ddd3..73be11198 100644 --- a/src/commands/replay/index.test.ts +++ b/src/commands/replay/index.test.ts @@ -131,3 +131,47 @@ describe('replay command interface', () => { expect(testRequest.options).not.toHaveProperty('reportJunit'); }); }); + +describe('replay resume (ADR 0012 decision 4 / migration step 5)', () => { + test('reads --from/--plan-digest as resumeFrom/resumePlanDigest, replay only', () => { + expect( + replayCliReader(['./checkout.ad'], flags({ replayFrom: 3, replayPlanDigest: 'deadbeef' })), + ).toMatchObject({ + path: './checkout.ad', + resumeFrom: 3, + resumePlanDigest: 'deadbeef', + }); + }); + + test('test CLI reader never surfaces resume fields, even if the flags carry them', () => { + const input = testCliReader( + ['./suite.ad'], + flags({ replayFrom: 3, replayPlanDigest: 'deadbeef' } as never), + ); + expect(input).not.toHaveProperty('resumeFrom'); + expect(input).not.toHaveProperty('resumePlanDigest'); + }); + + test('writes resumeFrom/resumePlanDigest onto the daemon request as replayFrom/replayPlanDigest', () => { + expect( + replayDaemonWriter({ + path: './checkout.ad', + resumeFrom: 3, + resumePlanDigest: 'deadbeef', + }), + ).toMatchObject({ + command: 'replay', + positionals: ['./checkout.ad'], + options: { + replayFrom: 3, + replayPlanDigest: 'deadbeef', + }, + }); + }); + + test('test daemon writer never emits replayFrom/replayPlanDigest', () => { + const request = testDaemonWriter({ paths: ['./suite.ad'] }); + expect(request.options).not.toHaveProperty('replayFrom'); + expect(request.options).not.toHaveProperty('replayPlanDigest'); + }); +}); diff --git a/src/commands/replay/index.ts b/src/commands/replay/index.ts index 29c0241f7..5582c7a35 100644 --- a/src/commands/replay/index.ts +++ b/src/commands/replay/index.ts @@ -35,6 +35,13 @@ export const replayCommandMetadata = defineFieldCommandMetadata( backend: stringField(), maestro: booleanField(), env: stringArrayField(), + // ADR 0012 decision 4 / migration step 5: replay-only resume. Named + // `resumeFrom`/`resumePlanDigest` (not `from`/`planDigest`) because + // `from` already means a gesture's `PointInput` on `CommandInput` + // (shared flat type across every command). `test` deliberately has + // neither field — it must stay a full, deterministic suite run. + resumeFrom: integerField(), + resumePlanDigest: stringField(), }, ); @@ -73,7 +80,15 @@ const replayCliSchema = { summary: replayCommandDescription, positionalArgs: ['path'], allowsExtraPositionals: true, - allowedFlags: ['replayMaestro', 'replayExportFormat', ...REPLAY_FLAGS, 'timeoutMs', 'out'], + allowedFlags: [ + 'replayMaestro', + 'replayExportFormat', + ...REPLAY_FLAGS, + 'replayFrom', + 'replayPlanDigest', + 'timeoutMs', + 'out', + ], } as const satisfies CommandSchemaOverride; const testCliSchema = { @@ -104,6 +119,8 @@ export const replayCliReader: CliReader = (positionals, flags) => ({ update: flags.replayUpdate, backend: flags.replayMaestro ? 'maestro' : undefined, env: flags.replayEnv, + resumeFrom: flags.replayFrom, + resumePlanDigest: flags.replayPlanDigest, }); export const testCliReader: CliReader = (positionals, flags) => ({ @@ -128,6 +145,8 @@ export const replayDaemonWriter: DaemonWriter = (input) => replayBackend: readReplayBackend(input), replayEnv: input.env, replayShellEnv: collectReplayClientShellEnv(process.env), + replayFrom: input.resumeFrom, + replayPlanDigest: input.resumePlanDigest, }); export const testDaemonWriter: DaemonWriter = (input) => diff --git a/src/compat/__tests__/replay-input.test.ts b/src/compat/__tests__/replay-input.test.ts index 0ee6f52c2..e79f6f636 100644 --- a/src/compat/__tests__/replay-input.test.ts +++ b/src/compat/__tests__/replay-input.test.ts @@ -14,7 +14,6 @@ test('parseReplayInput routes compat replay scripts through the selected parser' { replayBackend: 'maestro' }, ); - assert.match(parsed.updateUnsupportedMessage ?? '', /Convert to \.ad/); assert.deepEqual( parsed.actions.map((action) => [action.command, action.positionals]), [ diff --git a/src/compat/replay-input.ts b/src/compat/replay-input.ts index 9680d46d4..e86f5d636 100644 --- a/src/compat/replay-input.ts +++ b/src/compat/replay-input.ts @@ -23,7 +23,6 @@ type ReplayCompatParser = { export type ParsedReplayInput = ParsedReplayScript & { metadata: ReplayScriptMetadata; - updateUnsupportedMessage?: string; }; type ReplayInputParseOptions = { @@ -41,9 +40,6 @@ const REPLAY_COMPAT_PARSERS: Record = { }, }; -const COMPAT_UPDATE_UNSUPPORTED_MESSAGE = - 'replay -u is not supported for compat flow input. Convert to .ad first, then update that replay file.'; - export function parseReplayInput( script: string, flags: CommandFlags | undefined, @@ -51,14 +47,11 @@ export function parseReplayInput( ): ParsedReplayInput { const compatParser = readReplayCompatParser(flags); if (compatParser) { - return { - ...compatParser.parse(script, { - ...options, - platform: flags?.platform, - env: readReplayCompatEnv(flags), - }), - updateUnsupportedMessage: COMPAT_UPDATE_UNSUPPORTED_MESSAGE, - }; + return compatParser.parse(script, { + ...options, + platform: flags?.platform, + env: readReplayCompatEnv(flags), + }); } return { diff --git a/src/contracts/cli-flags.ts b/src/contracts/cli-flags.ts index af0010342..e150158f7 100644 --- a/src/contracts/cli-flags.ts +++ b/src/contracts/cli-flags.ts @@ -122,6 +122,8 @@ export type CliFlags = CloudProviderProfileFields & replayExportFormat?: 'maestro'; replayEnv?: string[]; replayShellEnv?: Record; + replayFrom?: number; + replayPlanDigest?: string; failFast?: boolean; timeoutMs?: number; retries?: number; diff --git a/src/daemon/handlers/__tests__/replay-heal.test.ts b/src/daemon/handlers/__tests__/replay-heal.test.ts deleted file mode 100644 index 3aeb2a11e..000000000 --- a/src/daemon/handlers/__tests__/replay-heal.test.ts +++ /dev/null @@ -1,905 +0,0 @@ -import { test, expect, vi, beforeEach } from 'vitest'; - -vi.mock('../../../core/dispatch.ts', async (importOriginal) => { - const actual = await importOriginal(); - return { ...actual, dispatchCommand: vi.fn(async () => ({})), resolveTargetDevice: vi.fn() }; -}); -vi.mock('../../device-ready.ts', () => ({ ensureDeviceReady: vi.fn(async () => {}) })); - -import fs from 'node:fs'; -import os from 'node:os'; -import path from 'node:path'; -import type { CommandFlags } from '../../../core/dispatch.ts'; -import { handleSessionCommands } from '../session.ts'; -import { healReplayAction } from '../session-replay-heal.ts'; -import { SessionStore } from '../../session-store.ts'; -import type { DaemonRequest, DaemonResponse, SessionAction } from '../../types.ts'; -import type { DeviceInfo } from '../../../kernel/device.ts'; -import { dispatchCommand, resolveTargetDevice } from '../../../core/dispatch.ts'; -import { ensureDeviceReady } from '../../device-ready.ts'; -import { makeIosSession } from '../../../__tests__/test-utils/session-factories.ts'; - -const mockDispatchCommand = vi.mocked(dispatchCommand); -const mockResolveTargetDevice = vi.mocked(resolveTargetDevice); -const mockEnsureDeviceReady = vi.mocked(ensureDeviceReady); - -beforeEach(() => { - mockDispatchCommand.mockReset(); - mockDispatchCommand.mockResolvedValue({}); - mockResolveTargetDevice.mockReset(); - mockEnsureDeviceReady.mockReset(); - mockEnsureDeviceReady.mockResolvedValue(undefined); -}); - -function makeSession(name: string) { - return makeIosSession(name, { appBundleId: 'com.example.app' }); -} - -function writeReplayFile(filePath: string, action: SessionAction) { - const args = action.positionals.map((value) => JSON.stringify(value)).join(' '); - fs.writeFileSync(filePath, `${action.command}${args.length > 0 ? ` ${args}` : ''}\n`); -} - -function readReplaySelector(filePath: string, command: string): string { - const lines = fs - .readFileSync(filePath, 'utf8') - .split(/\r?\n/) - .map((line) => line.trim()) - .filter((line) => line.length > 0); - const line = lines.find((entry) => entry.startsWith(`${command} `) || entry === command); - if (!line) return ''; - const args = tokenizeReplayLine(line).slice(1); - if (command === 'is') { - return args[1] ?? ''; - } - return args[0] ?? ''; -} - -function tokenizeReplayLine(line: string): string[] { - const tokens: string[] = []; - let cursor = 0; - while (cursor < line.length) { - while (cursor < line.length && /\s/.test(line[cursor]!)) { - cursor += 1; - } - if (cursor >= line.length) break; - if (line[cursor] === '"') { - let end = cursor + 1; - let escaped = false; - while (end < line.length) { - const char = line[end]; - if (char === '"' && !escaped) break; - escaped = char === '\\' && !escaped; - if (char !== '\\') escaped = false; - end += 1; - } - if (end >= line.length) { - throw new Error(`Invalid replay script line: ${line}`); - } - tokens.push(JSON.parse(line.slice(cursor, end + 1)) as string); - cursor = end + 1; - continue; - } - let end = cursor; - while (end < line.length && !/\s/.test(line[end]!)) { - end += 1; - } - tokens.push(line.slice(cursor, end)); - cursor = end; - } - return tokens; -} - -test('replay heal snapshot refresh clears stale scoped snapshot source', async () => { - const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-heal-scope-source-')); - const sessionsDir = path.join(tempRoot, 'sessions'); - const sessionStore = new SessionStore(sessionsDir); - const sessionName = 'heal-scope-source-session'; - const session = makeSession(sessionName); - session.snapshotScopeSource = { - nodes: [ - { - ref: 'e1', - index: 0, - depth: 0, - type: 'Button', - label: 'Stale button', - }, - ], - createdAt: Date.now(), - backend: 'xctest', - }; - sessionStore.set(sessionName, session); - - mockDispatchCommand.mockResolvedValue({ - nodes: [ - { - index: 0, - depth: 0, - type: 'Button', - label: 'Continue', - rect: { x: 0, y: 0, width: 100, height: 44 }, - hittable: true, - }, - ], - truncated: false, - backend: 'xctest', - }); - - const healed = await healReplayAction({ - action: { - ts: Date.now(), - command: 'click', - positionals: ['label="Continue"'], - flags: {}, - result: {}, - }, - sessionName, - logPath: '/tmp/replay.log', - sessionStore, - }); - - expect(healed?.positionals[0]).toContain('label="Continue"'); - expect(sessionStore.get(sessionName)?.snapshotScopeSource).toBeUndefined(); -}); - -test('replay heal rewrites longpress selector and preserves duration', async () => { - const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-heal-longpress-')); - const sessionsDir = path.join(tempRoot, 'sessions'); - const sessionStore = new SessionStore(sessionsDir); - const sessionName = 'heal-longpress-session'; - sessionStore.set(sessionName, makeSession(sessionName)); - - mockDispatchCommand.mockResolvedValue({ - nodes: [ - { - index: 0, - depth: 0, - type: 'XCUIElementTypeStaticText', - label: 'Last message', - identifier: 'message_last', - rect: { x: 0, y: 0, width: 100, height: 44 }, - hittable: true, - }, - ], - truncated: false, - backend: 'xctest', - }); - - const healed = await healReplayAction({ - action: { - ts: Date.now(), - command: 'longpress', - positionals: ['id="old_message" || label="Last message"', '800'], - flags: {}, - result: { selectorChain: ['id="old_message"', 'label="Last message"'], durationMs: 800 }, - }, - sessionName, - logPath: '/tmp/replay.log', - sessionStore, - }); - - expect(healed?.positionals[0]).toContain('message_last'); - expect(healed?.positionals[1]).toBe('800'); -}); - -test('replay heal uses canonical iOS snapshot presentation', async () => { - const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-heal-ios-row-')); - const sessionsDir = path.join(tempRoot, 'sessions'); - const sessionStore = new SessionStore(sessionsDir); - const sessionName = 'heal-ios-row-session'; - sessionStore.set(sessionName, makeSession(sessionName)); - const rowRect = { x: 16, y: 293, width: 370, height: 52 }; - - mockDispatchCommand.mockResolvedValue({ - nodes: [ - { index: 0, depth: 0, type: 'Application', label: 'Settings' }, - { index: 1, depth: 1, parentIndex: 0, type: 'CollectionView' }, - { index: 2, depth: 2, parentIndex: 1, type: 'Cell', label: 'General', rect: rowRect }, - { index: 3, depth: 3, parentIndex: 2, type: 'Other', label: 'General', rect: rowRect }, - { - index: 4, - depth: 4, - parentIndex: 3, - type: 'Button', - label: 'General', - identifier: 'com.apple.settings.general', - rect: rowRect, - }, - { index: 5, depth: 5, parentIndex: 4, type: 'StaticText', label: 'General', rect: rowRect }, - ], - truncated: false, - backend: 'xctest', - }); - - const healed = await healReplayAction({ - action: { - ts: Date.now(), - command: 'click', - positionals: ['label="General"'], - flags: {}, - result: { selectorChain: ['label="General"'] }, - }, - sessionName, - logPath: '/tmp/replay.log', - sessionStore, - }); - - expect(healed?.positionals[0]).toContain('com.apple.settings.general'); - expect(sessionStore.get(sessionName)?.snapshot?.nodes.map((node) => node.type)).toEqual([ - 'Application', - 'CollectionView', - 'Cell', - ]); -}); - -test('replay --update heals selector and rewrites replay file', async () => { - const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-heal-')); - const sessionsDir = path.join(tempRoot, 'sessions'); - const replayPath = path.join(tempRoot, 'replay.ad'); - const sessionStore = new SessionStore(sessionsDir); - const sessionName = 'heal-session'; - sessionStore.set(sessionName, makeSession(sessionName)); - - writeReplayFile(replayPath, { - ts: Date.now(), - command: 'click', - positionals: ['id="old_continue" || label="Continue"'], - flags: {}, - result: {}, - }); - - const invokeCalls: string[] = []; - const invoke = async (request: DaemonRequest): Promise => { - if (request.command !== 'click') { - return { - ok: false, - error: { code: 'INVALID_ARGS', message: `unexpected command ${request.command}` }, - }; - } - const selector = request.positionals?.[0] ?? ''; - invokeCalls.push(selector); - if (selector.includes('old_continue')) { - return { ok: false, error: { code: 'COMMAND_FAILED', message: 'selector no longer exists' } }; - } - if (selector.includes('auth_continue')) { - return { ok: true, data: { clicked: true } }; - } - return { ok: false, error: { code: 'COMMAND_FAILED', message: 'unexpected selector' } }; - }; - - let snapshotDispatchCalls = 0; - mockDispatchCommand.mockImplementation( - async ( - _device: DeviceInfo, - command: string, - _positionals: string[], - _out?: string, - _context?: CommandFlags, - ): Promise | void> => { - if (command !== 'snapshot') { - throw new Error(`unexpected dispatch command: ${command}`); - } - snapshotDispatchCalls += 1; - return { - nodes: [ - { - index: 0, - type: 'XCUIElementTypeButton', - label: 'Continue', - identifier: 'auth_continue', - rect: { x: 10, y: 10, width: 100, height: 44 }, - enabled: true, - hittable: true, - }, - ], - truncated: false, - backend: 'xctest', - }; - }, - ); - - const response = await handleSessionCommands({ - req: { - token: 't', - session: sessionName, - command: 'replay', - positionals: [replayPath], - flags: { replayUpdate: true }, - }, - sessionName, - logPath: path.join(tempRoot, 'daemon.log'), - sessionStore, - invoke, - }); - - expect(response).toBeTruthy(); - expect(response!.ok).toBe(true); - if (response!.ok) { - expect(response!.data?.healed).toBe(1); - expect(response!.data?.replayed).toBe(1); - } - expect(snapshotDispatchCalls).toBe(1); - expect(invokeCalls.length).toBe(2); - expect(invokeCalls[0]).toContain('old_continue'); - expect(invokeCalls[1]).toContain('auth_continue'); - const rewrittenSelector = readReplaySelector(replayPath, 'click'); - expect(rewrittenSelector).toContain('auth_continue'); - expect(rewrittenSelector).not.toContain('old_continue'); -}); - -test('replay tolerates legacy snapshot --backend and strips it on rewrite', async () => { - const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-legacy-backend-')); - const sessionsDir = path.join(tempRoot, 'sessions'); - const replayPath = path.join(tempRoot, 'replay.ad'); - const sessionStore = new SessionStore(sessionsDir); - const sessionName = 'legacy-backend-session'; - sessionStore.set(sessionName, makeSession(sessionName)); - fs.writeFileSync( - replayPath, - [ - 'snapshot -i --backend xctest', - 'click "id=\\"old_continue\\" || label=\\"Continue\\""', - '', - ].join('\n'), - ); - - const invoke = async (request: DaemonRequest): Promise => { - if (request.command === 'snapshot') { - return { ok: true, data: { nodes: [] } }; - } - if (request.command === 'click') { - const selector = request.positionals?.[0] ?? ''; - if (selector.includes('old_continue')) { - return { - ok: false, - error: { code: 'COMMAND_FAILED', message: 'selector no longer exists' }, - }; - } - if (selector.includes('auth_continue')) { - return { ok: true, data: { clicked: true } }; - } - } - return { - ok: false, - error: { code: 'INVALID_ARGS', message: `unexpected command ${request.command}` }, - }; - }; - - mockDispatchCommand.mockResolvedValue({ - nodes: [ - { - index: 0, - type: 'XCUIElementTypeButton', - label: 'Continue', - identifier: 'auth_continue', - rect: { x: 10, y: 10, width: 100, height: 44 }, - enabled: true, - hittable: true, - }, - ], - truncated: false, - backend: 'xctest', - }); - - const response = await handleSessionCommands({ - req: { - token: 't', - session: sessionName, - command: 'replay', - positionals: [replayPath], - flags: { replayUpdate: true }, - }, - sessionName, - logPath: path.join(tempRoot, 'daemon.log'), - sessionStore, - invoke, - }); - - expect(response).toBeTruthy(); - expect(response!.ok).toBe(true); - const rewritten = fs.readFileSync(replayPath, 'utf8'); - expect(rewritten).toMatch(/^snapshot -i$/m); - expect(rewritten).not.toMatch(/--backend/); -}); - -test('replay without --update does not heal or rewrite', async () => { - const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-noheal-')); - const sessionsDir = path.join(tempRoot, 'sessions'); - const replayPath = path.join(tempRoot, 'replay.ad'); - const sessionStore = new SessionStore(sessionsDir); - const sessionName = 'noheal-session'; - sessionStore.set(sessionName, makeSession(sessionName)); - - writeReplayFile(replayPath, { - ts: Date.now(), - command: 'click', - positionals: ['id="old_continue" || label="Continue"'], - flags: {}, - result: {}, - }); - const originalPayload = fs.readFileSync(replayPath, 'utf8'); - - const invoke = async (_request: DaemonRequest): Promise => { - return { - ok: false, - error: { - code: 'COMMAND_FAILED', - message: 'selector no longer exists', - hint: 'update selector', - diagnosticId: 'diag-replay-1', - logPath: '/tmp/diag-replay-1.ndjson', - }, - }; - }; - - const response = await handleSessionCommands({ - req: { - token: 't', - session: sessionName, - command: 'replay', - positionals: [replayPath], - flags: {}, - }, - sessionName, - logPath: path.join(tempRoot, 'daemon.log'), - sessionStore, - invoke, - }); - - expect(response).toBeTruthy(); - expect(response!.ok).toBe(false); - if (!response!.ok) { - expect(response!.error.message).toMatch(/Replay failed at step 1/); - expect(response!.error.details?.step).toBe(1); - expect(response!.error.details?.action).toBe('click'); - expect(response!.error.hint).toBe('update selector'); - expect(response!.error.diagnosticId).toBe('diag-replay-1'); - expect(response!.error.logPath).toBe('/tmp/diag-replay-1.ndjson'); - } - // No --update, so healReplayAction never runs (0 of its own capture calls) — - // but the divergence report's single shared capture (screen digest + - // suggestion re-resolution, ADR 0012 migration step 2) still fires - // unconditionally on failure. - expect(mockDispatchCommand).toHaveBeenCalledTimes(1); - expect(fs.readFileSync(replayPath, 'utf8')).toBe(originalPayload); -}); - -test('replay --update skips malformed selector candidates and preserves replay error context', async () => { - const tempRoot = fs.mkdtempSync( - path.join(os.tmpdir(), 'agent-device-replay-malformed-candidate-'), - ); - const sessionsDir = path.join(tempRoot, 'sessions'); - const replayPath = path.join(tempRoot, 'replay.ad'); - const sessionStore = new SessionStore(sessionsDir); - const sessionName = 'malformed-candidate-session'; - sessionStore.set(sessionName, makeSession(sessionName)); - - writeReplayFile(replayPath, { - ts: Date.now(), - command: 'click', - positionals: ['id="old_continue" ||'], - flags: {}, - result: {}, - }); - - const response = await handleSessionCommands({ - req: { - token: 't', - session: sessionName, - command: 'replay', - positionals: [replayPath], - flags: { replayUpdate: true }, - }, - sessionName, - logPath: path.join(tempRoot, 'daemon.log'), - sessionStore, - invoke: async () => ({ - ok: false, - error: { code: 'COMMAND_FAILED', message: 'selector stale' }, - }), - }); - - expect(response).toBeTruthy(); - expect(response!.ok).toBe(false); - if (!response!.ok) { - // ADR 0012 migration step 2: the wire-level code is now REPLAY_DIVERGENCE; - // the malformed candidate's original COMMAND_FAILED lives in cause. - expect(response!.error.code).toBe('REPLAY_DIVERGENCE'); - expect(response!.error.message).toMatch(/Replay failed at step 1/); - expect(response!.error.details?.step).toBe(1); - expect(response!.error.details?.action).toBe('click'); - const divergence = response!.error.details?.divergence as - | { cause: { code: string; message: string } } - | undefined; - expect(divergence?.cause.code).toBe('COMMAND_FAILED'); - // The malformed selector candidate is still skipped by suggestion - // ranking (tryParseSelectorChain rejects it), so no suggestion is produced. - expect((divergence as unknown as { suggestions: unknown[] })?.suggestions).toEqual([]); - } - // The divergence report's single shared capture still fires on every - // failure, even when heal itself would have skipped (malformed candidate) — - // ADR 0012 migration step 2 behavior, not a heal regression. - expect(mockDispatchCommand).toHaveBeenCalledTimes(1); - expect(fs.readFileSync(replayPath, 'utf8')).toBe('click "id=\\"old_continue\\" ||"\n'); -}); - -test('replay --update heals selector in is command', async () => { - const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-heal-is-')); - const sessionsDir = path.join(tempRoot, 'sessions'); - const replayPath = path.join(tempRoot, 'replay.ad'); - const sessionStore = new SessionStore(sessionsDir); - const sessionName = 'heal-is-session'; - sessionStore.set(sessionName, makeSession(sessionName)); - - writeReplayFile(replayPath, { - ts: Date.now(), - command: 'is', - positionals: ['visible', 'id="old_continue" || label="Continue"'], - flags: {}, - result: {}, - }); - - const invoke = async (request: DaemonRequest): Promise => { - if (request.command !== 'is') { - return { - ok: false, - error: { code: 'INVALID_ARGS', message: `unexpected command ${request.command}` }, - }; - } - const selector = request.positionals?.[1] ?? ''; - if (selector.includes('old_continue')) { - return { ok: false, error: { code: 'COMMAND_FAILED', message: 'selector stale' } }; - } - if (selector.includes('auth_continue')) { - return { ok: true, data: { predicate: 'visible', pass: true } }; - } - return { ok: false, error: { code: 'COMMAND_FAILED', message: 'unexpected selector' } }; - }; - - mockDispatchCommand.mockResolvedValue({ - nodes: [ - { - index: 0, - type: 'XCUIElementTypeButton', - label: 'Continue', - identifier: 'auth_continue', - rect: { x: 10, y: 10, width: 100, height: 44 }, - enabled: true, - hittable: true, - }, - ], - truncated: false, - backend: 'xctest', - }); - - const response = await handleSessionCommands({ - req: { - token: 't', - session: sessionName, - command: 'replay', - positionals: [replayPath], - flags: { replayUpdate: true }, - }, - sessionName, - logPath: path.join(tempRoot, 'daemon.log'), - sessionStore, - invoke, - }); - - expect(response).toBeTruthy(); - expect(response!.ok).toBe(true); - if (response!.ok) { - expect(response!.data?.healed).toBe(1); - } - const rewrittenSelector = readReplaySelector(replayPath, 'is'); - expect(rewrittenSelector).toContain('auth_continue'); -}); - -test('replay --update does not heal clicks from stored ref labels alone', async () => { - const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-heal-ref-label-')); - const sessionsDir = path.join(tempRoot, 'sessions'); - const replayPath = path.join(tempRoot, 'replay.ad'); - const sessionStore = new SessionStore(sessionsDir); - const sessionName = 'heal-ref-label-session'; - sessionStore.set(sessionName, makeSession(sessionName)); - - // @refs are never selector candidates; this guards against reintroducing - // fallback healing from the stored replay label. - fs.writeFileSync(replayPath, 'click @e1 "Continue"\n'); - const originalPayload = fs.readFileSync(replayPath, 'utf8'); - - const invokeCalls: string[] = []; - const invoke = async (request: DaemonRequest): Promise => { - if (request.command !== 'click') { - return { - ok: false, - error: { code: 'INVALID_ARGS', message: `unexpected command ${request.command}` }, - }; - } - const target = request.positionals?.[0] ?? ''; - invokeCalls.push(target); - return { ok: false, error: { code: 'COMMAND_FAILED', message: 'missing ref target' } }; - }; - - mockDispatchCommand.mockResolvedValue({ - nodes: [ - { - index: 0, - type: 'XCUIElementTypeButton', - label: 'Continue', - identifier: 'auth_continue', - rect: { x: 10, y: 10, width: 100, height: 44 }, - enabled: true, - hittable: true, - }, - ], - truncated: false, - backend: 'xctest', - }); - - const response = await handleSessionCommands({ - req: { - token: 't', - session: sessionName, - command: 'replay', - positionals: [replayPath], - flags: { replayUpdate: true }, - }, - sessionName, - logPath: path.join(tempRoot, 'daemon.log'), - sessionStore, - invoke, - }); - - expect(response).toBeTruthy(); - expect(response!.ok).toBe(false); - if (!response!.ok) { - expect(response!.error.message).toMatch(/Replay failed at step 1/); - expect(response!.error.details?.step).toBe(1); - expect(response!.error.details?.action).toBe('click'); - const divergence = response!.error.details?.divergence as - | { suggestions: unknown[] } - | undefined; - // @refs are never selector candidates, so no suggestion is produced — - // same non-healing guarantee this test title asserts, now for suggestions. - expect(divergence?.suggestions).toEqual([]); - } - // The divergence report's single shared capture still fires on every - // failure (ADR 0012 migration step 2) even though no suggestion candidate - // exists for an @ref-targeted action. - expect(mockDispatchCommand).toHaveBeenCalledTimes(1); - expect(invokeCalls).toEqual(['@e1']); - expect(fs.readFileSync(replayPath, 'utf8')).toBe(originalPayload); -}); - -test('replay --update does not heal numeric get text drift from snapshot text alone', async () => { - const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-heal-get-numeric-')); - const sessionsDir = path.join(tempRoot, 'sessions'); - const replayPath = path.join(tempRoot, 'replay.ad'); - const sessionStore = new SessionStore(sessionsDir); - const sessionName = 'heal-get-numeric-session'; - sessionStore.set(sessionName, makeSession(sessionName)); - - writeReplayFile(replayPath, { - ts: Date.now(), - command: 'get', - positionals: ['text', 'role="statictext" label="2" || label="2"'], - flags: {}, - result: {}, - }); - const originalPayload = fs.readFileSync(replayPath, 'utf8'); - - const invokeCalls: string[] = []; - const invoke = async (request: DaemonRequest): Promise => { - if (request.command !== 'get') { - return { - ok: false, - error: { code: 'INVALID_ARGS', message: `unexpected command ${request.command}` }, - }; - } - const selector = request.positionals?.[1] ?? ''; - invokeCalls.push(selector); - return { ok: false, error: { code: 'COMMAND_FAILED', message: 'selector stale' } }; - }; - - mockDispatchCommand.mockResolvedValue({ - nodes: [ - { - index: 0, - type: 'XCUIElementTypeStaticText', - label: '20', - rect: { x: 0, y: 100, width: 100, height: 24 }, - enabled: true, - hittable: true, - }, - { - index: 1, - type: 'XCUIElementTypeStaticText', - label: 'Version: 0.84.0', - rect: { x: 0, y: 200, width: 220, height: 17 }, - enabled: true, - hittable: true, - }, - ], - truncated: false, - backend: 'xctest', - }); - - const response = await handleSessionCommands({ - req: { - token: 't', - session: sessionName, - command: 'replay', - positionals: [replayPath], - flags: { replayUpdate: true }, - }, - sessionName, - logPath: path.join(tempRoot, 'daemon.log'), - sessionStore, - invoke, - }); - - expect(response).toBeTruthy(); - expect(response!.ok).toBe(false); - if (!response!.ok) { - expect(response!.error.message).toMatch(/Replay failed at step 1/); - expect(response!.error.details?.step).toBe(1); - expect(response!.error.details?.action).toBe('get'); - const divergence = response!.error.details?.divergence as - | { suggestions: unknown[] } - | undefined; - // Same non-healing guarantee as heal itself: a numeric label drift is not - // isolated by re-resolving the recorded selector, so no suggestion either. - expect(divergence?.suggestions).toEqual([]); - } - // 1 capture from healReplayAction's own (unsuccessful) attempt, plus the - // divergence report's single shared capture (ADR 0012 migration step 2) - // once heal gives up and the step fails. - expect(mockDispatchCommand).toHaveBeenCalledTimes(2); - expect(invokeCalls.length).toBe(1); - expect(fs.readFileSync(replayPath, 'utf8')).toBe(originalPayload); -}); - -test('replay --update heals selector in press command and preserves press series flags', async () => { - const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-heal-press-')); - const sessionsDir = path.join(tempRoot, 'sessions'); - const replayPath = path.join(tempRoot, 'replay.ad'); - const sessionStore = new SessionStore(sessionsDir); - const sessionName = 'heal-press-session'; - sessionStore.set(sessionName, makeSession(sessionName)); - fs.writeFileSync( - replayPath, - 'press "id=\\"old_continue\\" || label=\\"Continue\\"" --count 3 --interval-ms 1 --double-tap\n', - ); - - const invokeCalls: DaemonRequest[] = []; - const invoke = async (request: DaemonRequest): Promise => { - if (request.command !== 'press') { - return { - ok: false, - error: { code: 'INVALID_ARGS', message: `unexpected command ${request.command}` }, - }; - } - invokeCalls.push(request); - const selector = request.positionals?.[0] ?? ''; - if (selector.includes('old_continue')) { - return { ok: false, error: { code: 'COMMAND_FAILED', message: 'selector no longer exists' } }; - } - if (selector.includes('auth_continue')) { - return { ok: true, data: { pressed: true } }; - } - return { ok: false, error: { code: 'COMMAND_FAILED', message: 'unexpected selector' } }; - }; - - mockDispatchCommand.mockResolvedValue({ - nodes: [ - { - index: 0, - type: 'XCUIElementTypeButton', - label: 'Continue', - identifier: 'auth_continue', - rect: { x: 10, y: 10, width: 100, height: 44 }, - enabled: true, - hittable: true, - }, - ], - truncated: false, - backend: 'xctest', - }); - - const response = await handleSessionCommands({ - req: { - token: 't', - session: sessionName, - command: 'replay', - positionals: [replayPath], - flags: { replayUpdate: true }, - }, - sessionName, - logPath: path.join(tempRoot, 'daemon.log'), - sessionStore, - invoke, - }); - - expect(response).toBeTruthy(); - expect(response!.ok).toBe(true); - if (response!.ok) { - expect(response!.data?.healed).toBe(1); - expect(response!.data?.replayed).toBe(1); - } - expect(invokeCalls.length).toBe(2); - expect(invokeCalls[0]?.flags?.count).toBe(3); - expect(invokeCalls[0]?.flags?.intervalMs).toBe(1); - expect(invokeCalls[0]?.flags?.doubleTap).toBe(true); - const updatedLine = fs - .readFileSync(replayPath, 'utf8') - .split(/\r?\n/) - .find((line) => line.startsWith('press ')); - expect(updatedLine).toBeTruthy(); - const tokens = tokenizeReplayLine(updatedLine!); - expect(tokens[1]).toContain('auth_continue'); - expect(tokens.slice(2)).toEqual(['--count', '3', '--interval-ms', '1', '--double-tap']); -}); - -test('replay rejects legacy JSON payload files', async () => { - const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-json-rejected-')); - const sessionsDir = path.join(tempRoot, 'sessions'); - const replayPath = path.join(tempRoot, 'replay.json'); - const sessionStore = new SessionStore(sessionsDir); - const sessionName = 'json-rejected-session'; - sessionStore.set(sessionName, makeSession(sessionName)); - fs.writeFileSync(replayPath, JSON.stringify({ optimizedActions: [] }, null, 2)); - - const response = await handleSessionCommands({ - req: { - token: 't', - session: sessionName, - command: 'replay', - positionals: [replayPath], - flags: {}, - }, - sessionName, - logPath: path.join(tempRoot, 'daemon.log'), - sessionStore, - invoke: async () => ({ ok: true, data: {} }), - }); - - expect(response).toBeTruthy(); - expect(response!.ok).toBe(false); - if (!response!.ok) { - expect(response!.error.code).toBe('INVALID_ARGS'); - expect(response!.error.message).toMatch(/\.ad script files/); - } -}); - -test('replay rejects malformed .ad lines with unclosed quotes', async () => { - const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-invalid-ad-')); - const sessionsDir = path.join(tempRoot, 'sessions'); - const replayPath = path.join(tempRoot, 'replay.ad'); - const sessionStore = new SessionStore(sessionsDir); - const sessionName = 'invalid-ad-session'; - sessionStore.set(sessionName, makeSession(sessionName)); - fs.writeFileSync(replayPath, 'click "id=\\"broken\\"\n'); - - const response = await handleSessionCommands({ - req: { - token: 't', - session: sessionName, - command: 'replay', - positionals: [replayPath], - flags: {}, - }, - sessionName, - logPath: path.join(tempRoot, 'daemon.log'), - sessionStore, - invoke: async () => ({ ok: true, data: {} }), - }); - - expect(response).toBeTruthy(); - expect(response!.ok).toBe(false); - if (!response!.ok) { - expect(response!.error.code).toBe('INVALID_ARGS'); - expect(response!.error.message).toMatch(/Invalid replay script line/); - } -}); diff --git a/src/daemon/handlers/__tests__/session-replay-divergence.test.ts b/src/daemon/handlers/__tests__/session-replay-divergence.test.ts new file mode 100644 index 000000000..acf535811 --- /dev/null +++ b/src/daemon/handlers/__tests__/session-replay-divergence.test.ts @@ -0,0 +1,71 @@ +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { beforeEach, expect, test, vi } from 'vitest'; + +vi.mock('../../../core/dispatch.ts', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, dispatchCommand: vi.fn(async () => ({})), resolveTargetDevice: vi.fn() }; +}); + +import { dispatchCommand } from '../../../core/dispatch.ts'; +import { makeIosSession } from '../../../__tests__/test-utils/session-factories.ts'; +import { SessionStore } from '../../session-store.ts'; +import { buildReplayFailureDivergence } from '../session-replay-divergence.ts'; + +const mockDispatchCommand = vi.mocked(dispatchCommand); + +beforeEach(() => { + mockDispatchCommand.mockReset(); + mockDispatchCommand.mockResolvedValue({}); +}); + +test('buildReplayFailureDivergence dedupes suggestions using the strongest basis', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-suggest-dedupe-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); + + mockDispatchCommand.mockResolvedValue({ + nodes: [ + { + index: 0, + depth: 0, + type: 'Button', + label: 'Save', + identifier: 'save', + rect: { x: 0, y: 0, width: 100, height: 44 }, + hittable: true, + }, + ], + truncated: false, + backend: 'xctest', + }); + + const action = { + ts: 0, + command: 'click', + positionals: ['label="Save"'], + flags: {}, + result: { selectorChain: ['label="Save"', 'id="save"'] }, + }; + const divergence = await buildReplayFailureDivergence({ + error: { code: 'COMMAND_FAILED', message: 'not hittable' }, + action, + index: 0, + sourcePath: path.join(root, 'flow.ad'), + sourceLine: 1, + session: sessionStore.get(sessionName), + sessionName, + sessionStore, + logPath: path.join(root, 'daemon.log'), + responseLevel: 'default', + planActions: [action], + planDigest: 'test-plan-digest', + }); + + expect(divergence.suggestionCount).toBe(1); + expect(divergence.suggestions).toHaveLength(1); + expect(divergence.suggestions[0]?.ref).toBe('e1'); + expect(divergence.suggestions[0]?.basis).toBe('id'); +}); diff --git a/src/daemon/handlers/__tests__/session-replay-resume.test.ts b/src/daemon/handlers/__tests__/session-replay-resume.test.ts new file mode 100644 index 000000000..4746e94d9 --- /dev/null +++ b/src/daemon/handlers/__tests__/session-replay-resume.test.ts @@ -0,0 +1,166 @@ +import assert from 'node:assert/strict'; +import { test } from 'vitest'; +import { + buildReplayDivergenceResume, + evaluateReplayResumePreflight, +} from '../session-replay-resume.ts'; +import type { SessionAction } from '../../types.ts'; + +function action(overrides: Partial = {}): SessionAction { + return { ts: 0, command: 'click', positionals: ['label="Save"'], flags: {}, ...overrides }; +} + +// --- from === 1: nothing skipped, always allowed --- + +test('from 1 is always allowed, even when step 1 itself is control flow', () => { + const actions: SessionAction[] = [ + action({ + command: 'back', + positionals: [], + replayControl: { kind: 'retry', maxRetries: 2, actions: [action({ command: 'back' })] }, + }), + action({ command: 'click' }), + ]; + assert.deepEqual(evaluateReplayResumePreflight({ from: 1, actions }), { allowed: true }); +}); + +// --- plain steps: always resumable --- + +test('resuming after only plain, non-control-flow steps is allowed', () => { + const actions: SessionAction[] = [ + action({ command: 'open', positionals: ['Demo'] }), + action({ command: 'click', positionals: ['label="Continue"'] }), + action({ command: 'click', positionals: ['label="Save"'] }), + ]; + assert.deepEqual(evaluateReplayResumePreflight({ from: 3, actions }), { allowed: true }); +}); + +// --- control flow in the skipped range --- + +test('rejects when a skipped step is a retry block', () => { + const actions: SessionAction[] = [ + action({ + command: 'back', + positionals: [], + replayControl: { kind: 'retry', maxRetries: 2, actions: [action({ command: 'back' })] }, + }), + action({ command: 'click' }), + ]; + const result = evaluateReplayResumePreflight({ from: 2, actions }); + assert.equal(result.allowed, false); + if (result.allowed) return; + assert.match(result.reason, /control flow/); + assert.match(result.reason, /retry/); +}); + +test('rejects when a skipped step is a maestroRunFlowWhen block', () => { + const actions: SessionAction[] = [ + action({ + command: 'back', + positionals: [], + replayControl: { + kind: 'maestroRunFlowWhen', + mode: 'visible', + selector: 'label="Continue"', + actions: [action({ command: 'back' })], + }, + }), + action({ command: 'click' }), + ]; + const result = evaluateReplayResumePreflight({ from: 2, actions }); + assert.equal(result.allowed, false); + if (result.allowed) return; + assert.match(result.reason, /maestroRunFlowWhen/); +}); + +// --- control flow AS the resume target --- + +test('rejects when the resume target itself is a control-flow block', () => { + const actions: SessionAction[] = [ + action({ command: 'open', positionals: ['Demo'] }), + action({ + command: 'back', + positionals: [], + replayControl: { kind: 'retry', maxRetries: 1, actions: [action({ command: 'back' })] }, + }), + ]; + const result = evaluateReplayResumePreflight({ from: 2, actions }); + assert.equal(result.allowed, false); + if (result.allowed) return; + assert.match(result.reason, /cannot be safely resumed into/); +}); + +// --- outputEnv-producing skipped steps --- + +test('rejects when a skipped step can produce outputEnv values (maestro runScript)', () => { + const actions: SessionAction[] = [ + action({ command: '__maestroRunScript', positionals: ['./setup.js'] }), + action({ command: 'click' }), + ]; + const result = evaluateReplayResumePreflight({ from: 2, actions }); + assert.equal(result.allowed, false); + if (result.allowed) return; + assert.match(result.reason, /outputEnv/); +}); + +test('an outputEnv-producing step AT the resume target itself (not skipped) is fine', () => { + const actions: SessionAction[] = [ + action({ command: 'open', positionals: ['Demo'] }), + action({ command: '__maestroRunScript', positionals: ['./setup.js'] }), + ]; + // Resuming AT the runScript step re-executes it (it is not skipped), so its + // outputEnv is produced fresh, not assumed from a prior run. + assert.deepEqual(evaluateReplayResumePreflight({ from: 2, actions }), { allowed: true }); +}); + +// --- same child index recurring under different parents / repeated plan indices --- +// (Resume addressing is purely by top-level plan index; two structurally +// distinct occurrences of the same command are still distinguished correctly +// because each occupies its own array slot.) + +test('expanded repeats occupy distinct, independently addressable plan indices', () => { + const actions: SessionAction[] = [ + action({ command: 'click', positionals: ['label="Item"'] }), + action({ command: 'click', positionals: ['label="Item"'] }), + action({ command: 'click', positionals: ['label="Item"'] }), + ]; + // Resuming at the 3rd occurrence skips the first two identical-looking + // steps; neither is control-flow or outputEnv-producing, so it is allowed. + assert.deepEqual(evaluateReplayResumePreflight({ from: 3, actions }), { allowed: true }); +}); + +// --- buildReplayDivergenceResume: report-facing wrapper --- + +test('buildReplayDivergenceResume reports allowed:true with from/planDigest for a safe failed step', () => { + const actions: SessionAction[] = [ + action({ command: 'open', positionals: ['Demo'] }), + action({ command: 'click', positionals: ['label="Save"'] }), + ]; + const resume = buildReplayDivergenceResume({ + failedIndex: 2, + actions, + planDigest: 'abc123', + }); + assert.deepEqual(resume, { allowed: true, from: 2, planDigest: 'abc123' }); +}); + +test('buildReplayDivergenceResume reports allowed:false with from/planDigest/reason when unsafe', () => { + const actions: SessionAction[] = [ + action({ + command: 'back', + positionals: [], + replayControl: { kind: 'retry', maxRetries: 1, actions: [action({ command: 'back' })] }, + }), + action({ command: 'click' }), + ]; + const resume = buildReplayDivergenceResume({ + failedIndex: 2, + actions, + planDigest: 'abc123', + }); + assert.equal(resume.allowed, false); + assert.equal(resume.from, 2); + assert.equal(resume.planDigest, 'abc123'); + if (resume.allowed) return; + assert.ok(resume.reason.length > 0); +}); diff --git a/src/daemon/handlers/__tests__/session-replay-runtime-artifacts.test.ts b/src/daemon/handlers/__tests__/session-replay-runtime-artifacts.test.ts new file mode 100644 index 000000000..bb61cc35a --- /dev/null +++ b/src/daemon/handlers/__tests__/session-replay-runtime-artifacts.test.ts @@ -0,0 +1,51 @@ +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { test } from 'vitest'; +import { collectReplayActionArtifactPaths } from '../session-replay-runtime-artifacts.ts'; + +test('collectReplayActionArtifactPaths includes existing failed action artifacts', () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-artifacts-')); + const snapshotPath = path.join(root, 'failure-snapshot.txt'); + fs.writeFileSync(snapshotPath, 'snapshot'); + + const paths = collectReplayActionArtifactPaths({ + ok: false, + error: { + code: 'COMMAND_FAILED', + message: 'assertion failed', + details: { + artifactPaths: [snapshotPath, path.join(root, 'missing.txt')], + }, + }, + }); + + assert.deepEqual(paths, [snapshotPath]); +}); + +test('collectReplayActionArtifactPaths collects top-level and nested successful artifacts', () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-success-artifacts-')); + const pathValue = path.join(root, 'path.txt'); + const outPath = path.join(root, 'out.txt'); + const localPath = path.join(root, 'local.txt'); + const nestedPath = path.join(root, 'nested.txt'); + for (const artifactPath of [pathValue, outPath, localPath, nestedPath]) { + fs.writeFileSync(artifactPath, 'artifact'); + } + + const paths = collectReplayActionArtifactPaths({ + ok: true, + data: { + path: pathValue, + outPath, + artifacts: [ + { field: 'preferred', localPath, path: nestedPath }, + { field: 'nested', path: nestedPath }, + { field: 'missing', path: path.join(root, 'missing.txt') }, + ], + }, + }); + + assert.deepEqual(paths, [pathValue, outPath, localPath, nestedPath]); +}); diff --git a/src/daemon/handlers/__tests__/session-replay-runtime-failure-response.test.ts b/src/daemon/handlers/__tests__/session-replay-runtime-failure-response.test.ts new file mode 100644 index 000000000..15c962727 --- /dev/null +++ b/src/daemon/handlers/__tests__/session-replay-runtime-failure-response.test.ts @@ -0,0 +1,219 @@ +import { test, expect, vi, beforeEach } from 'vitest'; + +vi.mock('../../../core/dispatch.ts', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, dispatchCommand: vi.fn(async () => ({})), resolveTargetDevice: vi.fn() }; +}); +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { runReplayScriptFile } from '../session-replay-runtime.ts'; +import { SessionStore } from '../../session-store.ts'; +import { dispatchCommand } from '../../../core/dispatch.ts'; +import { makeIosSession } from '../../../__tests__/test-utils/session-factories.ts'; +import { + baseReplayRequest as baseReq, + writeReplayFile, +} from './session-replay-runtime.fixtures.ts'; + +const mockDispatchCommand = vi.mocked(dispatchCommand); + +beforeEach(() => { + mockDispatchCommand.mockReset(); + mockDispatchCommand.mockResolvedValue({}); +}); +test('divergence cause and action strings pass through the central redactor at construction', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-redact-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName)); + const filePath = writeReplayFile(root, ['click "Save"']); + mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); + + const response = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath] }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async () => ({ + ok: false, + error: { + code: 'COMMAND_FAILED', + message: 'request rejected: api_key=sk-live-abc123def456 invalid', + }, + }), + }); + + expect(response.ok).toBe(false); + if (response.ok) return; + const divergence = response.error.details?.divergence as Record; + const cause = divergence.cause as { message: string }; + expect(cause.message).not.toContain('sk-live-abc123def456'); + expect(cause.message).toContain('api_key=[REDACTED]'); +}); + +// --- Blocker 1: fill text must NEVER appear in the divergence output --- + +test('a fill divergence never serializes the typed text at any response level', async () => { + const sentinel = 'SuperSecretPassword-do-not-leak-12345'; + for (const level of ['digest', 'default', 'full'] as const) { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-fill-leak-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); + const filePath = writeReplayFile(root, [`fill 'label="Email"' ${JSON.stringify(sentinel)}`]); + // Selector miss forces the divergence on the fill step; the failure + // message is a realistic selector error, not an echo of the typed text. + mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); + + const response = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath], meta: { responseLevel: level } }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async (req) => { + if (req.command === 'fill') { + // A REAL fill-verification failure carries the entered text in + // details.expected (unmasked fields do, by the fill-diagnostics + // contract) — the divergence transport must strip it categorically. + return { + ok: false, + error: { + code: 'COMMAND_FAILED', + message: 'Android fill verification failed', + details: { + expected: sentinel, + actual: sentinel.slice(0, 10), + failureReason: 'text_mismatch', + }, + }, + }; + } + return { ok: true, data: {} }; + }, + }); + + expect(response.ok).toBe(false); + if (response.ok) return; + const serializedDivergence = JSON.stringify(response.error.details?.divergence); + expect(serializedDivergence).not.toContain(sentinel); + // The WHOLE public error (flat details incl. the cause's own + // details.expected/actual, positionals, message) must not leak it either. + expect(JSON.stringify(response.error)).not.toContain(sentinel); + expect(JSON.stringify(response.error)).not.toContain(sentinel.slice(0, 10)); + // The action label still names the field, with the text categorically hidden. + const divergence = response.error.details?.divergence as { action: string }; + expect(divergence.action).toContain(''); + expect(divergence.action).toContain('Email'); + } +}); + +// --- Blocker 3a: capture-error screen hint is sanitized --- + +test('a capture-failed screen hint redacts a secret in the capture error', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-screen-redact-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); + const filePath = writeReplayFile(root, ['click "Save"']); + // The post-failure snapshot capture throws with a secret-bearing message. + mockDispatchCommand.mockRejectedValue(new Error('snapshot failed: api_key=sk-live-abc123def456')); + + const response = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath] }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async () => ({ + ok: false, + error: { code: 'COMMAND_FAILED', message: 'Selector did not match' }, + }), + }); + + expect(response.ok).toBe(false); + if (response.ok) return; + const divergence = response.error.details?.divergence as { + screen: { state: string; reason?: string; hint?: string }; + }; + expect(divergence.screen.state).toBe('unavailable'); + expect(divergence.screen.reason).toBe('capture-failed'); + expect(divergence.screen.hint).not.toContain('sk-live-abc123def456'); + expect(divergence.screen.hint).toContain('[REDACTED]'); +}); + +// --- Expanded replay variables are never serialized (ADR 0012) --- + +test('an expanded ${VAR} value echoed by a selector error never reaches the public divergence', async () => { + const sentinel = 'ExpandedVarSecret-98765-do-not-leak'; + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-var-leak-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); + const filePath = writeReplayFile(root, ['press label="${SECRET}"']); + mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); + + const response = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath], flags: { replayEnv: [`SECRET=${sentinel}`] } }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async (req) => { + if (req.command === 'press') { + // A real selector miss echoes the RESOLVED (expanded) selector. + return { + ok: false, + error: { + code: 'COMMAND_FAILED', + message: `Selector did not match: ${req.positionals?.[0] ?? ''}`, + hint: `Run find "${sentinel}" for contains matching.`, + }, + }; + } + return { ok: true, data: {} }; + }, + }); + + expect(response.ok).toBe(false); + if (response.ok) return; + // The expanded value appears nowhere in the whole public error. + expect(JSON.stringify(response.error)).not.toContain(sentinel); + // The scrub is a marker replacement, not a drop: the caller still sees + // WHICH variable the selector interpolated. + const divergence = response.error.details?.divergence as { + cause: { message: string; hint?: string }; + }; + expect(divergence.cause.message).toContain(''); + expect(divergence.cause.hint).toContain(''); + expect(response.error.message).toContain(''); +}); + +test('an expanded built-in AD_DEVICE_ID never reaches the public divergence', async () => { + const deviceId = 'BuiltInDeviceId-486b3d4c-8f92-4dc0-b5c6-unique'; + const sessionName = 'static-session-context'; + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-builtin-var-leak-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); + const filePath = writeReplayFile(root, ['press label="${AD_DEVICE_ID}"']); + mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); + + const response = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath], flags: { serial: deviceId } }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async (req) => ({ + ok: false, + error: { + code: 'COMMAND_FAILED', + message: `Device ${req.positionals?.[0] ?? ''} failed in ${sessionName} context.`, + }, + }), + }); + + expect(response.ok).toBe(false); + if (response.ok) return; + expect(JSON.stringify(response.error)).not.toContain(deviceId); + expect(response.error.message).toContain(''); + // AD_SESSION was not expanded, so matching static text remains readable. + expect(response.error.message).toContain(sessionName); +}); diff --git a/src/daemon/handlers/__tests__/session-replay-runtime-failure.test.ts b/src/daemon/handlers/__tests__/session-replay-runtime-failure.test.ts new file mode 100644 index 000000000..423beb7c6 --- /dev/null +++ b/src/daemon/handlers/__tests__/session-replay-runtime-failure.test.ts @@ -0,0 +1,364 @@ +import { test, expect, vi, beforeEach } from 'vitest'; + +vi.mock('../../../core/dispatch.ts', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, dispatchCommand: vi.fn(async () => ({})), resolveTargetDevice: vi.fn() }; +}); +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { runReplayScriptFile } from '../session-replay-runtime.ts'; +import { SessionStore } from '../../session-store.ts'; +import type { DaemonResponse } from '../../types.ts'; +import { dispatchCommand } from '../../../core/dispatch.ts'; +import { makeIosSession } from '../../../__tests__/test-utils/session-factories.ts'; +import { + baseReplayRequest as baseReq, + writeReplayFile, +} from './session-replay-runtime.fixtures.ts'; + +const mockDispatchCommand = vi.mocked(dispatchCommand); + +beforeEach(() => { + mockDispatchCommand.mockReset(); + mockDispatchCommand.mockResolvedValue({}); +}); +test('a failing replay step returns REPLAY_DIVERGENCE with cause preserved and correct step provenance', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); + const filePath = writeReplayFile(root, ['open "Demo"', 'click "Save"']); + + // The post-failure screen digest capture (and the suggestions re-resolution + // capture) both go through dispatchCommand('snapshot', ...); with no real + // device backend in a unit test, this throws, so screen must degrade to + // 'unavailable' rather than masking the original replay cause. + mockDispatchCommand.mockImplementation(async (_device, command) => { + if (command === 'snapshot') throw new Error('no device runner available'); + return { ok: true }; + }); + + const response = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath] }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async (req) => { + if (req.command === 'open') return { ok: true, data: { session: sessionName } }; + if (req.command === 'click') { + return { + ok: false, + error: { code: 'COMMAND_FAILED', message: 'Selector did not match', hint: 'Run find.' }, + }; + } + throw new Error(`unexpected command ${req.command}`); + }, + }); + + expect(response.ok).toBe(false); + if (response.ok) return; + expect(response.error.code).toBe('REPLAY_DIVERGENCE'); + // The legacy flat fields survive additively for existing consumers. + expect(response.error.details?.step).toBe(2); + expect(response.error.details?.action).toBe('click'); + + const divergence = response.error.details?.divergence as Record; + expect(divergence.version).toBe(1); + expect(divergence.kind).toBe('action-failure'); + const step = divergence.step as { index: number; source: { path: string; line: number } }; + expect(step.index).toBe(2); + expect(step.source.path).toBe(filePath); + expect(step.source.line).toBe(2); + + const cause = divergence.cause as { code: string; message: string; hint?: string }; + expect(cause.code).toBe('COMMAND_FAILED'); + expect(cause.message).toBe('Selector did not match'); + expect(cause.hint).toBe('Run find.'); + + const screen = divergence.screen as { state: string; reason?: string }; + expect(screen.state).toBe('unavailable'); + expect(screen.reason).toBe('capture-failed'); + + expect(divergence.suggestions).toEqual([]); + expect(divergence.suggestionCount).toBe(0); + const resume = divergence.resume as { allowed: boolean; from: number; planDigest: string }; + // Step 1 ("open") is a plain action with no control flow and no outputEnv + // production, so resuming at the failed step (2) is safe. + expect(resume).toEqual({ allowed: true, from: 2, planDigest: expect.any(String) }); + expect(resume.planDigest).toMatch(/^[0-9a-f]{64}$/); +}); + +test('a normalized nested failure preserves typed recovery signals on REPLAY_DIVERGENCE', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-recovery-signals-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); + const filePath = writeReplayFile(root, ['click "Save"']); + mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); + + const response = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath] }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + // This shape is already normalized: normalizeError lifted both signals + // out of details before the replay wrapper receives it. + invoke: async () => ({ + ok: false, + error: { + code: 'DEVICE_IN_USE', + message: 'The device is temporarily leased.', + retriable: true, + supportedOn: 'ios', + }, + }), + }); + + expect(response.ok).toBe(false); + if (response.ok) return; + expect(response.error.code).toBe('REPLAY_DIVERGENCE'); + expect(response.error.retriable).toBe(true); + expect(response.error.supportedOn).toBe('ios'); +}); + +test('a failing replay step captures an available screen digest with blessed refs', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-screen-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); + const filePath = writeReplayFile(root, ['click "Save"']); + + mockDispatchCommand.mockResolvedValue({ + nodes: [ + { + ref: 'e1', + index: 0, + depth: 0, + type: 'Button', + label: 'Cancel', + rect: { x: 0, y: 0, width: 100, height: 44 }, + hittable: true, + }, + ], + truncated: false, + backend: 'xctest', + }); + + const response = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath] }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async () => ({ + ok: false, + error: { code: 'COMMAND_FAILED', message: 'Selector did not match' }, + }), + }); + + expect(response.ok).toBe(false); + if (response.ok) return; + const divergence = response.error.details?.divergence as Record; + const screen = divergence.screen as { + state: string; + refsGeneration: number; + refs: Array<{ ref: string; role: string; label?: string }>; + }; + expect(screen.state).toBe('available'); + expect(typeof screen.refsGeneration).toBe('number'); + expect(screen.refs).toEqual([{ ref: 'e1', role: 'button', label: 'Cancel' }]); +}); + +test('a failing replay step ranks a re-resolved suggestion when the recorded selector still matches', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-suggest-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); + // The recorded selector is label-only, so it still structurally matches a + // node in the fresh capture even though the underlying tap failed (e.g. the + // node moved off-screen or was momentarily not hittable) — the exact class + // heal could recover, now surfaced as a read-only suggestion instead. + const filePath = writeReplayFile(root, ['click label="Save"']); + + mockDispatchCommand.mockResolvedValue({ + nodes: [ + { + index: 0, + depth: 0, + type: 'Button', + label: 'Save', + rect: { x: 0, y: 0, width: 100, height: 44 }, + hittable: true, + }, + ], + truncated: false, + backend: 'xctest', + }); + + const response = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath] }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async () => ({ + ok: false, + error: { code: 'COMMAND_FAILED', message: 'not hittable' }, + }), + }); + + expect(response.ok).toBe(false); + if (response.ok) return; + const divergence = response.error.details?.divergence as Record; + const suggestions = divergence.suggestions as Array<{ + selector: string; + basis: string; + ref?: string; + }>; + expect(divergence.suggestionCount).toBe(1); + expect(suggestions).toHaveLength(1); + expect(suggestions[0]?.ref).toBe('e1'); + expect(suggestions[0]?.basis).toBe('label'); +}); + +test('divergence screen never masks the original cause when the session already closed', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-no-session-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + // Intentionally no session stored: simulates the session closing mid-replay. + const filePath = writeReplayFile(root, ['click "Save"']); + + const response: DaemonResponse = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath] }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async () => ({ + ok: false, + error: { code: 'COMMAND_FAILED', message: 'session closed mid-replay' }, + }), + }); + + expect(response.ok).toBe(false); + if (response.ok) return; + expect(response.error.code).toBe('REPLAY_DIVERGENCE'); + const divergence = response.error.details?.divergence as Record; + const cause = divergence.cause as { message: string }; + expect(cause.message).toBe('session closed mid-replay'); + const screen = divergence.screen as { state: string; reason?: string }; + expect(screen.state).toBe('unavailable'); + expect(screen.reason).toBe('no-session'); +}); + +// --- Control-flow-wrapped include provenance (reviewer probe scenario) --- +// +// The RN suite's own launch include is retry-wrapped, so the single most +// common real failure site (a launch wait timeout inside the include) must +// report the INCLUDE's file+line, not the wrapping `retry:`/`runFlow.when:` +// line in the root flow. Regression for the leak where replayControl.actions +// kept the transient replaySource field but the runtime never consulted it. + +function writeMaestroInclude(root: string): string { + const childPath = path.join(root, 'child.yaml'); + fs.writeFileSync( + childPath, + ['appId: com.callstack.agentdevicelab', '---', '- back', ''].join('\n'), + ); + return childPath; +} + +test('a failure inside a retry-wrapped runFlow include reports the include file and line', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-retry-provenance-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName)); + const childPath = writeMaestroInclude(root); + const mainPath = path.join(root, 'main.yaml'); + fs.writeFileSync( + mainPath, + [ + 'appId: com.callstack.agentdevicelab', + '---', + '- retry:', + ' maxRetries: 1', + ' commands:', + ' - runFlow:', + ' file: child.yaml', + '', + ].join('\n'), + ); + mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); + + const response = await runReplayScriptFile({ + req: baseReq({ positionals: [mainPath], flags: { replayBackend: 'maestro' } }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async (req) => { + if (req.command === 'back') { + return { ok: false, error: { code: 'COMMAND_FAILED', message: 'back failed' } }; + } + return { ok: true, data: {} }; + }, + }); + + expect(response.ok).toBe(false); + if (response.ok) return; + expect(response.error.code).toBe('REPLAY_DIVERGENCE'); + const divergence = response.error.details?.divergence as Record; + const step = divergence.step as { index: number; source: { path: string; line: number } }; + // Plan index 1: the retry wrapper is one executable-plan step; the source + // names the failing NESTED action inside the include, not the retry: line. + expect(step.index).toBe(1); + expect(step.source.path).toBe(childPath); + expect(step.source.line).toBe(3); + // The transport-internal provenance marker is stripped from the flat details. + expect(response.error.details?.replaySource).toBeUndefined(); +}); + +test('a failure inside a runtime runFlow.when-wrapped include reports the include file and line', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-when-provenance-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName)); + const childPath = writeMaestroInclude(root); + const mainPath = path.join(root, 'main.yaml'); + fs.writeFileSync( + mainPath, + [ + 'appId: com.callstack.agentdevicelab', + '---', + '- runFlow:', + ' file: child.yaml', + ' when:', + ' notVisible: Continue', + '', + ].join('\n'), + ); + mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); + + const response = await runReplayScriptFile({ + req: baseReq({ positionals: [mainPath], flags: { replayBackend: 'maestro' } }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async (req) => { + // The notVisible condition captures a snapshot; an empty tree means the + // selector is absent, so the wrapped steps run. + if (req.command === 'snapshot') return { ok: true, data: { nodes: [] } }; + if (req.command === 'back') { + return { ok: false, error: { code: 'COMMAND_FAILED', message: 'back failed' } }; + } + return { ok: true, data: {} }; + }, + }); + + expect(response.ok).toBe(false); + if (response.ok) return; + expect(response.error.code).toBe('REPLAY_DIVERGENCE'); + const divergence = response.error.details?.divergence as Record; + const step = divergence.step as { index: number; source: { path: string; line: number } }; + expect(step.index).toBe(1); + expect(step.source.path).toBe(childPath); + expect(step.source.line).toBe(3); + expect(response.error.details?.replaySource).toBeUndefined(); +}); diff --git a/src/daemon/handlers/__tests__/session-replay-runtime-plan.test.ts b/src/daemon/handlers/__tests__/session-replay-runtime-plan.test.ts new file mode 100644 index 000000000..1fa90ad78 --- /dev/null +++ b/src/daemon/handlers/__tests__/session-replay-runtime-plan.test.ts @@ -0,0 +1,314 @@ +import { test, expect, vi, beforeEach } from 'vitest'; + +vi.mock('../../../core/dispatch.ts', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, dispatchCommand: vi.fn(async () => ({})), resolveTargetDevice: vi.fn() }; +}); + +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { runReplayScriptFile } from '../session-replay-runtime.ts'; +import { SessionStore } from '../../session-store.ts'; +import { dispatchCommand } from '../../../core/dispatch.ts'; +import { makeIosSession } from '../../../__tests__/test-utils/session-factories.ts'; +import { + baseReplayRequest as baseReq, + writeReplayFile, +} from './session-replay-runtime.fixtures.ts'; + +const mockDispatchCommand = vi.mocked(dispatchCommand); + +beforeEach(() => { + mockDispatchCommand.mockReset(); + mockDispatchCommand.mockResolvedValue({}); +}); +test('resume skips steps 1..from-1 without invoking them and executes only from the reported step', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-resume-skip-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName)); + const filePath = writeReplayFile(root, [ + 'open "Demo"', + 'click label="Continue"', + 'click label="Save"', + ]); + + // First attempt: step 3 fails, capturing a real resume report. + const firstAttempt = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath] }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async (req) => { + if (req.command === 'click' && req.positionals?.[0] === 'label="Save"') { + return { ok: false, error: { code: 'COMMAND_FAILED', message: 'not hittable' } }; + } + return { ok: true, data: {} }; + }, + }); + expect(firstAttempt.ok).toBe(false); + if (firstAttempt.ok) return; + const divergence = firstAttempt.error.details?.divergence as { + resume: { allowed: boolean; from: number; planDigest: string }; + }; + expect(divergence.resume.allowed).toBe(true); + expect(divergence.resume.from).toBe(3); + + // Second attempt: repair app state, resume at the reported step. Steps 1-2 + // must never be invoked — the mock throws if they are. + const invokedCommands: string[] = []; + const resumedAttempt = await runReplayScriptFile({ + req: baseReq({ + positionals: [filePath], + flags: { replayFrom: divergence.resume.from, replayPlanDigest: divergence.resume.planDigest }, + }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async (req) => { + invokedCommands.push(`${req.command} ${req.positionals?.[0] ?? ''}`.trim()); + if (req.command === 'open' || req.positionals?.[0] === 'label="Continue"') { + throw new Error('resume must not re-invoke a skipped step'); + } + return { ok: true, data: {} }; + }, + }); + + expect(resumedAttempt.ok).toBe(true); + if (!resumedAttempt.ok) return; + expect(invokedCommands).toEqual(['click label="Save"']); + const data = resumedAttempt.data as { replayed: number }; + expect(data.replayed).toBe(1); +}); + +test('resume requires both --from and --plan-digest together', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-resume-pair-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName)); + const filePath = writeReplayFile(root, ['open "Demo"', 'click "Save"']); + + const fromOnly = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath], flags: { replayFrom: 2 } }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async () => { + throw new Error('must not execute before the flag-pair preflight'); + }, + }); + expect(fromOnly.ok).toBe(false); + if (!fromOnly.ok) expect(fromOnly.error.code).toBe('INVALID_ARGS'); + + const digestOnly = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath], flags: { replayPlanDigest: 'deadbeef' } }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async () => { + throw new Error('must not execute before the flag-pair preflight'); + }, + }); + expect(digestOnly.ok).toBe(false); + if (!digestOnly.ok) expect(digestOnly.error.code).toBe('INVALID_ARGS'); +}); + +test('resume rejects an out-of-range --from before any action', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-resume-range-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName)); + const filePath = writeReplayFile(root, ['open "Demo"', 'click "Save"']); + + const response = await runReplayScriptFile({ + req: baseReq({ + positionals: [filePath], + flags: { replayFrom: 99, replayPlanDigest: 'deadbeef' }, + }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async () => { + throw new Error('must not execute an out-of-range resume'); + }, + }); + expect(response.ok).toBe(false); + if (response.ok) return; + expect(response.error.code).toBe('INVALID_ARGS'); + expect(response.error.message).toMatch(/out of range/); +}); + +test('resume rejects a stale --plan-digest after the script changed', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-resume-stale-digest-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName)); + const filePath = writeReplayFile(root, ['open "Demo"', 'click "Save"']); + + const firstAttempt = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath] }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async (req) => { + if (req.command === 'click') { + return { ok: false, error: { code: 'COMMAND_FAILED', message: 'not hittable' } }; + } + return { ok: true, data: {} }; + }, + }); + expect(firstAttempt.ok).toBe(false); + if (firstAttempt.ok) return; + const divergence = firstAttempt.error.details?.divergence as { + resume: { allowed: boolean; from: number; planDigest: string }; + }; + + // Edit the script (an extra step) before resuming: the digest must no + // longer match. + fs.writeFileSync(filePath, 'open "Demo"\nclick "Extra"\nclick "Save"\n'); + + const resumedAttempt = await runReplayScriptFile({ + req: baseReq({ + positionals: [filePath], + flags: { replayFrom: divergence.resume.from, replayPlanDigest: divergence.resume.planDigest }, + }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async () => { + throw new Error('must not execute on a plan-digest mismatch'); + }, + }); + expect(resumedAttempt.ok).toBe(false); + if (resumedAttempt.ok) return; + expect(resumedAttempt.error.code).toBe('INVALID_ARGS'); + expect(resumedAttempt.error.message).toMatch(/plan digest/); +}); + +test('resume rejects a digest from a different effective platform or target before any action', async () => { + const root = fs.mkdtempSync( + path.join(os.tmpdir(), 'agent-device-replay-resume-effective-target-'), + ); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName)); + const filePath = writeReplayFile(root, [ + 'context platform=android target=tv', + 'open "Demo"', + 'click "Save"', + ]); + const executionFlags = { platform: 'ios' as const, target: 'mobile' as const }; + + const firstAttempt = await runReplayScriptFile({ + req: baseReq({ positionals: [filePath], flags: executionFlags }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async (req) => + req.command === 'click' + ? { ok: false, error: { code: 'COMMAND_FAILED', message: 'not hittable' } } + : { ok: true, data: {} }, + }); + expect(firstAttempt.ok).toBe(false); + if (firstAttempt.ok) return; + const divergence = firstAttempt.error.details?.divergence as { + resume: { from: number; planDigest: string }; + }; + + for (const changedExecutionFlags of [ + { ...executionFlags, target: 'desktop' as const }, + { ...executionFlags, platform: 'android' as const }, + ]) { + const resumedAttempt = await runReplayScriptFile({ + req: baseReq({ + positionals: [filePath], + flags: { + ...changedExecutionFlags, + replayFrom: divergence.resume.from, + replayPlanDigest: divergence.resume.planDigest, + }, + }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async () => { + throw new Error('must not execute when effective replay target changed'); + }, + }); + + expect(resumedAttempt.ok).toBe(false); + if (!resumedAttempt.ok) { + expect(resumedAttempt.error.code).toBe('INVALID_ARGS'); + expect(resumedAttempt.error.message).toMatch(/plan digest/); + } + } +}); +test('resume rejects resuming past a retry-wrapped step in the skipped range', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-resume-control-flow-')); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + const sessionName = 'default'; + sessionStore.set(sessionName, makeIosSession(sessionName)); + const mainPath = path.join(root, 'main.yaml'); + fs.writeFileSync( + mainPath, + [ + 'appId: com.callstack.agentdevicelab', + '---', + '- retry:', + ' maxRetries: 1', + ' commands:', + ' - back', + '- back', + '', + ].join('\n'), + ); + + // `back` has no Maestro-specific runtime handling, so it reaches `invoke` + // directly — the retry block's nested `back` (1st call) succeeds; the + // top-level step-2 `back` (2nd call) fails. + let backCalls = 0; + const firstAttempt = await runReplayScriptFile({ + req: baseReq({ positionals: [mainPath], flags: { replayBackend: 'maestro' } }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async (req) => { + if (req.command !== 'back') return { ok: true, data: {} }; + backCalls += 1; + if (backCalls === 1) return { ok: true, data: {} }; + return { ok: false, error: { code: 'COMMAND_FAILED', message: 'back failed' } }; + }, + }); + expect(firstAttempt.ok).toBe(false); + if (firstAttempt.ok) return; + const divergence = firstAttempt.error.details?.divergence as { + resume: { allowed: boolean; from: number; planDigest: string; reason?: string }; + }; + // Step 2 (tapOn: Save) is the reported failure; resuming there means + // skipping step 1, the retry block. + expect(divergence.resume.from).toBe(2); + expect(divergence.resume.allowed).toBe(false); + expect(divergence.resume.reason).toMatch(/control flow/); + + const resumedAttempt = await runReplayScriptFile({ + req: baseReq({ + positionals: [mainPath], + flags: { + replayBackend: 'maestro', + replayFrom: divergence.resume.from, + replayPlanDigest: divergence.resume.planDigest, + }, + }), + sessionName, + logPath: path.join(root, 'daemon.log'), + sessionStore, + invoke: async () => { + throw new Error('must not execute a resume the preflight rejected'); + }, + }); + expect(resumedAttempt.ok).toBe(false); + if (resumedAttempt.ok) return; + expect(resumedAttempt.error.code).toBe('INVALID_ARGS'); + expect(resumedAttempt.error.message).toMatch(/control flow/); +}); diff --git a/src/daemon/handlers/__tests__/session-replay-runtime.fixtures.ts b/src/daemon/handlers/__tests__/session-replay-runtime.fixtures.ts new file mode 100644 index 000000000..f76f6c0f8 --- /dev/null +++ b/src/daemon/handlers/__tests__/session-replay-runtime.fixtures.ts @@ -0,0 +1,19 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import type { DaemonRequest } from '../../types.ts'; + +export function writeReplayFile(root: string, lines: string[]): string { + const filePath = path.join(root, 'flow.ad'); + fs.writeFileSync(filePath, `${lines.join('\n')}\n`); + return filePath; +} + +export function baseReplayRequest(overrides: Partial = {}): DaemonRequest { + return { + token: 'token', + session: 'default', + command: 'replay', + positionals: [], + ...overrides, + }; +} diff --git a/src/daemon/handlers/__tests__/session-replay-runtime.test.ts b/src/daemon/handlers/__tests__/session-replay-runtime.test.ts index e2076eded..79fc8e996 100644 --- a/src/daemon/handlers/__tests__/session-replay-runtime.test.ts +++ b/src/daemon/handlers/__tests__/session-replay-runtime.test.ts @@ -9,11 +9,13 @@ import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; import { runReplayScriptFile } from '../session-replay-runtime.ts'; -import { buildReplayFailureDivergence } from '../session-replay-divergence.ts'; import { SessionStore } from '../../session-store.ts'; -import type { DaemonRequest, DaemonResponse } from '../../types.ts'; import { dispatchCommand } from '../../../core/dispatch.ts'; import { makeIosSession } from '../../../__tests__/test-utils/session-factories.ts'; +import { + baseReplayRequest as baseReq, + writeReplayFile, +} from './session-replay-runtime.fixtures.ts'; const mockDispatchCommand = vi.mocked(dispatchCommand); @@ -21,176 +23,83 @@ beforeEach(() => { mockDispatchCommand.mockReset(); mockDispatchCommand.mockResolvedValue({}); }); - -function writeReplayFile(root: string, lines: string[]): string { - const filePath = path.join(root, 'flow.ad'); - fs.writeFileSync(filePath, `${lines.join('\n')}\n`); - return filePath; -} - -function baseReq(overrides: Partial = {}): DaemonRequest { - return { - token: 'token', - session: 'default', - command: 'replay', - positionals: [], - ...overrides, - }; -} - -test('a failing replay step returns REPLAY_DIVERGENCE with cause preserved and correct step provenance', async () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-')); +test('a successful replay prints one line with the step count and wall time', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-success-message-')); const sessionStore = new SessionStore(path.join(root, 'sessions')); const sessionName = 'default'; - sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); + sessionStore.set(sessionName, makeIosSession(sessionName)); const filePath = writeReplayFile(root, ['open "Demo"', 'click "Save"']); - // The post-failure screen digest capture (and the suggestions re-resolution - // capture) both go through dispatchCommand('snapshot', ...); with no real - // device backend in a unit test, this throws, so screen must degrade to - // 'unavailable' rather than masking the original replay cause. - mockDispatchCommand.mockImplementation(async (_device, command) => { - if (command === 'snapshot') throw new Error('no device runner available'); - return { ok: true }; - }); - const response = await runReplayScriptFile({ req: baseReq({ positionals: [filePath] }), sessionName, logPath: path.join(root, 'daemon.log'), sessionStore, - invoke: async (req) => { - if (req.command === 'open') return { ok: true, data: { session: sessionName } }; - if (req.command === 'click') { - return { - ok: false, - error: { code: 'COMMAND_FAILED', message: 'Selector did not match', hint: 'Run find.' }, - }; - } - throw new Error(`unexpected command ${req.command}`); - }, + invoke: async () => ({ ok: true, data: {} }), }); - expect(response.ok).toBe(false); - if (response.ok) return; - expect(response.error.code).toBe('REPLAY_DIVERGENCE'); - // The legacy flat fields survive additively for existing consumers. - expect(response.error.details?.step).toBe(2); - expect(response.error.details?.action).toBe('click'); - - const divergence = response.error.details?.divergence as Record; - expect(divergence.version).toBe(1); - expect(divergence.kind).toBe('action-failure'); - const step = divergence.step as { index: number; source: { path: string; line: number } }; - expect(step.index).toBe(2); - expect(step.source.path).toBe(filePath); - expect(step.source.line).toBe(2); - - const cause = divergence.cause as { code: string; message: string; hint?: string }; - expect(cause.code).toBe('COMMAND_FAILED'); - expect(cause.message).toBe('Selector did not match'); - expect(cause.hint).toBe('Run find.'); - - const screen = divergence.screen as { state: string; reason?: string }; - expect(screen.state).toBe('unavailable'); - expect(screen.reason).toBe('capture-failed'); - - expect(divergence.suggestions).toEqual([]); - expect(divergence.suggestionCount).toBe(0); - expect(divergence.resume).toEqual({ allowed: false, reason: 'resume not yet supported' }); + expect(response.ok).toBe(true); + if (!response.ok) return; + const data = response.data as { replayed: number; message: string }; + expect(data.replayed).toBe(2); + expect(data.message).toMatch(/^Replayed 2 steps in \d+\.\ds$/); }); - -test('a normalized nested failure preserves typed recovery signals on REPLAY_DIVERGENCE', async () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-recovery-signals-')); +test('replay rejects legacy JSON payload files', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-json-rejected-')); const sessionStore = new SessionStore(path.join(root, 'sessions')); const sessionName = 'default'; - sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); - const filePath = writeReplayFile(root, ['click "Save"']); - mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); + sessionStore.set(sessionName, makeIosSession(sessionName)); + const filePath = path.join(root, 'replay.json'); + fs.writeFileSync(filePath, JSON.stringify({ optimizedActions: [] }, null, 2)); const response = await runReplayScriptFile({ req: baseReq({ positionals: [filePath] }), sessionName, logPath: path.join(root, 'daemon.log'), sessionStore, - // This shape is already normalized: normalizeError lifted both signals - // out of details before the replay wrapper receives it. - invoke: async () => ({ - ok: false, - error: { - code: 'DEVICE_IN_USE', - message: 'The device is temporarily leased.', - retriable: true, - supportedOn: 'ios', - }, - }), + invoke: async () => ({ ok: true, data: {} }), }); expect(response.ok).toBe(false); if (response.ok) return; - expect(response.error.code).toBe('REPLAY_DIVERGENCE'); - expect(response.error.retriable).toBe(true); - expect(response.error.supportedOn).toBe('ios'); + expect(response.error.code).toBe('INVALID_ARGS'); + expect(response.error.message).toMatch(/\.ad script files/); }); -test('a failing replay step captures an available screen digest with blessed refs', async () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-screen-')); +test('replay rejects malformed .ad lines with unclosed quotes', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-invalid-ad-')); const sessionStore = new SessionStore(path.join(root, 'sessions')); const sessionName = 'default'; - sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); - const filePath = writeReplayFile(root, ['click "Save"']); - - mockDispatchCommand.mockResolvedValue({ - nodes: [ - { - ref: 'e1', - index: 0, - depth: 0, - type: 'Button', - label: 'Cancel', - rect: { x: 0, y: 0, width: 100, height: 44 }, - hittable: true, - }, - ], - truncated: false, - backend: 'xctest', - }); + sessionStore.set(sessionName, makeIosSession(sessionName)); + const filePath = writeReplayFile(root, ['click "id=\\"broken\\"']); const response = await runReplayScriptFile({ req: baseReq({ positionals: [filePath] }), sessionName, logPath: path.join(root, 'daemon.log'), sessionStore, - invoke: async () => ({ - ok: false, - error: { code: 'COMMAND_FAILED', message: 'Selector did not match' }, - }), + invoke: async () => ({ ok: true, data: {} }), }); expect(response.ok).toBe(false); if (response.ok) return; - const divergence = response.error.details?.divergence as Record; - const screen = divergence.screen as { - state: string; - refsGeneration: number; - refs: Array<{ ref: string; role: string; label?: string }>; - }; - expect(screen.state).toBe('available'); - expect(typeof screen.refsGeneration).toBe('number'); - expect(screen.refs).toEqual([{ ref: 'e1', role: 'button', label: 'Cancel' }]); + expect(response.error.code).toBe('INVALID_ARGS'); + expect(response.error.message).toMatch(/Invalid replay script line/); }); -test('a failing replay step ranks a re-resolved suggestion when the recorded selector still matches', async () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-suggest-')); +// --- ADR 0012 decision 1 / migration step 6: `--update` retirement --- + +test('--update never rewrites the .ad file, even when a re-resolvable suggestion exists', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-update-no-write-')); const sessionStore = new SessionStore(path.join(root, 'sessions')); const sessionName = 'default'; sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); - // The recorded selector is label-only, so it still structurally matches a - // node in the fresh capture even though the underlying tap failed (e.g. the - // node moved off-screen or was momentarily not hittable) — the exact class - // heal could recover, now surfaced as a read-only suggestion instead. const filePath = writeReplayFile(root, ['click label="Save"']); + const before = fs.readFileSync(filePath, 'utf8'); + const statBefore = fs.statSync(filePath); + // The recorded selector still structurally matches a fresh node — exactly + // the case the old heal-and-rewrite arm would have silently applied. mockDispatchCommand.mockResolvedValue({ nodes: [ { @@ -207,7 +116,7 @@ test('a failing replay step ranks a re-resolved suggestion when the recorded sel }); const response = await runReplayScriptFile({ - req: baseReq({ positionals: [filePath] }), + req: baseReq({ positionals: [filePath], flags: { replayUpdate: true } }), sessionName, logPath: path.join(root, 'daemon.log'), sessionStore, @@ -219,27 +128,26 @@ test('a failing replay step ranks a re-resolved suggestion when the recorded sel expect(response.ok).toBe(false); if (response.ok) return; - const divergence = response.error.details?.divergence as Record; - const suggestions = divergence.suggestions as Array<{ - selector: string; - basis: string; - ref?: string; - }>; - expect(divergence.suggestionCount).toBe(1); - expect(suggestions).toHaveLength(1); - expect(suggestions[0]?.ref).toBe('e1'); - expect(suggestions[0]?.basis).toBe('label'); + // The file on disk is byte-for-byte unchanged. + expect(fs.readFileSync(filePath, 'utf8')).toBe(before); + expect(fs.statSync(filePath).mtimeMs).toBe(statBefore.mtimeMs); + // The bounded suggestions the ADR mandates are still there — --update did + // not lose functionality, it lost the unattended rewrite. + const divergence = response.error.details?.divergence as { + suggestions: Array<{ selector: string; basis: string }>; + }; + expect(divergence.suggestions.length).toBeGreaterThan(0); }); -test('a successful replay prints one line with the step count and wall time', async () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-success-message-')); +test('a successful --update replay reports healed: 0 (heal is retired, not just quiet)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-update-healed-zero-')); const sessionStore = new SessionStore(path.join(root, 'sessions')); const sessionName = 'default'; sessionStore.set(sessionName, makeIosSession(sessionName)); const filePath = writeReplayFile(root, ['open "Demo"', 'click "Save"']); const response = await runReplayScriptFile({ - req: baseReq({ positionals: [filePath] }), + req: baseReq({ positionals: [filePath], flags: { replayUpdate: true } }), sessionName, logPath: path.join(root, 'daemon.log'), sessionStore, @@ -248,398 +156,45 @@ test('a successful replay prints one line with the step count and wall time', as expect(response.ok).toBe(true); if (!response.ok) return; - const data = response.data as { replayed: number; message: string }; - expect(data.replayed).toBe(2); - expect(data.message).toMatch(/^Replayed 2 steps in \d+\.\ds$/); -}); - -test('divergence screen never masks the original cause when the session already closed', async () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-no-session-')); - const sessionStore = new SessionStore(path.join(root, 'sessions')); - const sessionName = 'default'; - // Intentionally no session stored: simulates the session closing mid-replay. - const filePath = writeReplayFile(root, ['click "Save"']); - - const response: DaemonResponse = await runReplayScriptFile({ - req: baseReq({ positionals: [filePath] }), - sessionName, - logPath: path.join(root, 'daemon.log'), - sessionStore, - invoke: async () => ({ - ok: false, - error: { code: 'COMMAND_FAILED', message: 'session closed mid-replay' }, - }), - }); - - expect(response.ok).toBe(false); - if (response.ok) return; - expect(response.error.code).toBe('REPLAY_DIVERGENCE'); - const divergence = response.error.details?.divergence as Record; - const cause = divergence.cause as { message: string }; - expect(cause.message).toBe('session closed mid-replay'); - const screen = divergence.screen as { state: string; reason?: string }; - expect(screen.state).toBe('unavailable'); - expect(screen.reason).toBe('no-session'); -}); - -// --- Control-flow-wrapped include provenance (reviewer probe scenario) --- -// -// The RN suite's own launch include is retry-wrapped, so the single most -// common real failure site (a launch wait timeout inside the include) must -// report the INCLUDE's file+line, not the wrapping `retry:`/`runFlow.when:` -// line in the root flow. Regression for the leak where replayControl.actions -// kept the transient replaySource field but the runtime never consulted it. - -function writeMaestroInclude(root: string): string { - const childPath = path.join(root, 'child.yaml'); - fs.writeFileSync( - childPath, - ['appId: com.callstack.agentdevicelab', '---', '- back', ''].join('\n'), - ); - return childPath; -} - -test('a failure inside a retry-wrapped runFlow include reports the include file and line', async () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-retry-provenance-')); - const sessionStore = new SessionStore(path.join(root, 'sessions')); - const sessionName = 'default'; - sessionStore.set(sessionName, makeIosSession(sessionName)); - const childPath = writeMaestroInclude(root); - const mainPath = path.join(root, 'main.yaml'); - fs.writeFileSync( - mainPath, - [ - 'appId: com.callstack.agentdevicelab', - '---', - '- retry:', - ' maxRetries: 1', - ' commands:', - ' - runFlow:', - ' file: child.yaml', - '', - ].join('\n'), - ); - mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); - - const response = await runReplayScriptFile({ - req: baseReq({ positionals: [mainPath], flags: { replayBackend: 'maestro' } }), - sessionName, - logPath: path.join(root, 'daemon.log'), - sessionStore, - invoke: async (req) => { - if (req.command === 'back') { - return { ok: false, error: { code: 'COMMAND_FAILED', message: 'back failed' } }; - } - return { ok: true, data: {} }; - }, - }); - - expect(response.ok).toBe(false); - if (response.ok) return; - expect(response.error.code).toBe('REPLAY_DIVERGENCE'); - const divergence = response.error.details?.divergence as Record; - const step = divergence.step as { index: number; source: { path: string; line: number } }; - // Plan index 1: the retry wrapper is one executable-plan step; the source - // names the failing NESTED action inside the include, not the retry: line. - expect(step.index).toBe(1); - expect(step.source.path).toBe(childPath); - expect(step.source.line).toBe(3); - // The transport-internal provenance marker is stripped from the flat details. - expect(response.error.details?.replaySource).toBeUndefined(); + const data = response.data as { healed: number }; + expect(data.healed).toBe(0); }); -test('a failure inside a runtime runFlow.when-wrapped include reports the include file and line', async () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-when-provenance-')); +test('--update no longer refuses env directives (the guard existed only for rewrite safety)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-update-env-ok-')); const sessionStore = new SessionStore(path.join(root, 'sessions')); const sessionName = 'default'; sessionStore.set(sessionName, makeIosSession(sessionName)); - const childPath = writeMaestroInclude(root); - const mainPath = path.join(root, 'main.yaml'); - fs.writeFileSync( - mainPath, - [ - 'appId: com.callstack.agentdevicelab', - '---', - '- runFlow:', - ' file: child.yaml', - ' when:', - ' notVisible: Continue', - '', - ].join('\n'), - ); - mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); + const filePath = writeReplayFile(root, ['env NAME=World', 'open "Demo"']); const response = await runReplayScriptFile({ - req: baseReq({ positionals: [mainPath], flags: { replayBackend: 'maestro' } }), + req: baseReq({ positionals: [filePath], flags: { replayUpdate: true } }), sessionName, logPath: path.join(root, 'daemon.log'), sessionStore, - invoke: async (req) => { - // The notVisible condition captures a snapshot; an empty tree means the - // selector is absent, so the wrapped steps run. - if (req.command === 'snapshot') return { ok: true, data: { nodes: [] } }; - if (req.command === 'back') { - return { ok: false, error: { code: 'COMMAND_FAILED', message: 'back failed' } }; - } - return { ok: true, data: {} }; - }, + invoke: async () => ({ ok: true, data: {} }), }); - expect(response.ok).toBe(false); - if (response.ok) return; - expect(response.error.code).toBe('REPLAY_DIVERGENCE'); - const divergence = response.error.details?.divergence as Record; - const step = divergence.step as { index: number; source: { path: string; line: number } }; - expect(step.index).toBe(1); - expect(step.source.path).toBe(childPath); - expect(step.source.line).toBe(3); - expect(response.error.details?.replaySource).toBeUndefined(); + expect(response.ok).toBe(true); }); -test('divergence cause and action strings pass through the central redactor at construction', async () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-divergence-redact-')); +test('--update no longer refuses ${VAR} interpolation (the guard existed only for rewrite safety)', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-update-interp-ok-')); const sessionStore = new SessionStore(path.join(root, 'sessions')); const sessionName = 'default'; sessionStore.set(sessionName, makeIosSession(sessionName)); - const filePath = writeReplayFile(root, ['click "Save"']); - mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); - - const response = await runReplayScriptFile({ - req: baseReq({ positionals: [filePath] }), - sessionName, - logPath: path.join(root, 'daemon.log'), - sessionStore, - invoke: async () => ({ - ok: false, - error: { - code: 'COMMAND_FAILED', - message: 'request rejected: api_key=sk-live-abc123def456 invalid', - }, - }), - }); - - expect(response.ok).toBe(false); - if (response.ok) return; - const divergence = response.error.details?.divergence as Record; - const cause = divergence.cause as { message: string }; - expect(cause.message).not.toContain('sk-live-abc123def456'); - expect(cause.message).toContain('api_key=[REDACTED]'); -}); - -// --- Blocker 1: fill text must NEVER appear in the divergence output --- - -test('a fill divergence never serializes the typed text at any response level', async () => { - const sentinel = 'SuperSecretPassword-do-not-leak-12345'; - for (const level of ['digest', 'default', 'full'] as const) { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-fill-leak-')); - const sessionStore = new SessionStore(path.join(root, 'sessions')); - const sessionName = 'default'; - sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); - const filePath = writeReplayFile(root, [`fill 'label="Email"' ${JSON.stringify(sentinel)}`]); - // Selector miss forces the divergence on the fill step; the failure - // message is a realistic selector error, not an echo of the typed text. - mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); - - const response = await runReplayScriptFile({ - req: baseReq({ positionals: [filePath], meta: { responseLevel: level } }), - sessionName, - logPath: path.join(root, 'daemon.log'), - sessionStore, - invoke: async (req) => { - if (req.command === 'fill') { - // A REAL fill-verification failure carries the entered text in - // details.expected (unmasked fields do, by the fill-diagnostics - // contract) — the divergence transport must strip it categorically. - return { - ok: false, - error: { - code: 'COMMAND_FAILED', - message: 'Android fill verification failed', - details: { - expected: sentinel, - actual: sentinel.slice(0, 10), - failureReason: 'text_mismatch', - }, - }, - }; - } - return { ok: true, data: {} }; - }, - }); - - expect(response.ok).toBe(false); - if (response.ok) return; - const serializedDivergence = JSON.stringify(response.error.details?.divergence); - expect(serializedDivergence).not.toContain(sentinel); - // The WHOLE public error (flat details incl. the cause's own - // details.expected/actual, positionals, message) must not leak it either. - expect(JSON.stringify(response.error)).not.toContain(sentinel); - expect(JSON.stringify(response.error)).not.toContain(sentinel.slice(0, 10)); - // The action label still names the field, with the text categorically hidden. - const divergence = response.error.details?.divergence as { action: string }; - expect(divergence.action).toContain(''); - expect(divergence.action).toContain('Email'); - } -}); - -// --- Blocker 3b: suggestion dedupe keeps the STRONGEST basis per node --- - -test('a divergence dedupes suggestions by node and tags the strongest basis', async () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-suggest-dedupe-')); - const sessionStore = new SessionStore(path.join(root, 'sessions')); - const sessionName = 'default'; - sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); - - // A recorded click whose selectorChain lists a label-basis term FIRST and an - // id-basis term SECOND, both resolving to the same node. The suggestion must - // appear once, tagged with the stronger `id` basis (not the first-seen label). - mockDispatchCommand.mockResolvedValue({ - nodes: [ - { - index: 0, - depth: 0, - type: 'Button', - label: 'Save', - identifier: 'save', - rect: { x: 0, y: 0, width: 100, height: 44 }, - hittable: true, - }, - ], - truncated: false, - backend: 'xctest', - }); - - const divergence = await buildReplayFailureDivergence({ - error: { code: 'COMMAND_FAILED', message: 'not hittable' }, - action: { - ts: 0, - command: 'click', - positionals: ['label="Save"'], - flags: {}, - result: { selectorChain: ['label="Save"', 'id="save"'] }, - }, - index: 0, - sourcePath: path.join(root, 'flow.ad'), - sourceLine: 1, - session: sessionStore.get(sessionName), - sessionName, - sessionStore, - logPath: path.join(root, 'daemon.log'), - responseLevel: 'default', - }); - - expect(divergence.suggestionCount).toBe(1); - expect(divergence.suggestions).toHaveLength(1); - expect(divergence.suggestions[0]?.ref).toBe('e1'); - expect(divergence.suggestions[0]?.basis).toBe('id'); -}); - -// --- Blocker 3a: capture-error screen hint is sanitized --- - -test('a capture-failed screen hint redacts a secret in the capture error', async () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-screen-redact-')); - const sessionStore = new SessionStore(path.join(root, 'sessions')); - const sessionName = 'default'; - sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); - const filePath = writeReplayFile(root, ['click "Save"']); - // The post-failure snapshot capture throws with a secret-bearing message. - mockDispatchCommand.mockRejectedValue(new Error('snapshot failed: api_key=sk-live-abc123def456')); + const filePath = writeReplayFile(root, ['click label="${NAME}"']); const response = await runReplayScriptFile({ - req: baseReq({ positionals: [filePath] }), - sessionName, - logPath: path.join(root, 'daemon.log'), - sessionStore, - invoke: async () => ({ - ok: false, - error: { code: 'COMMAND_FAILED', message: 'Selector did not match' }, + req: baseReq({ + positionals: [filePath], + flags: { replayUpdate: true, replayEnv: ['NAME=World'] }, }), - }); - - expect(response.ok).toBe(false); - if (response.ok) return; - const divergence = response.error.details?.divergence as { - screen: { state: string; reason?: string; hint?: string }; - }; - expect(divergence.screen.state).toBe('unavailable'); - expect(divergence.screen.reason).toBe('capture-failed'); - expect(divergence.screen.hint).not.toContain('sk-live-abc123def456'); - expect(divergence.screen.hint).toContain('[REDACTED]'); -}); - -// --- Expanded replay variables are never serialized (ADR 0012) --- - -test('an expanded ${VAR} value echoed by a selector error never reaches the public divergence', async () => { - const sentinel = 'ExpandedVarSecret-98765-do-not-leak'; - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-var-leak-')); - const sessionStore = new SessionStore(path.join(root, 'sessions')); - const sessionName = 'default'; - sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); - const filePath = writeReplayFile(root, ['press label="${SECRET}"']); - mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); - - const response = await runReplayScriptFile({ - req: baseReq({ positionals: [filePath], flags: { replayEnv: [`SECRET=${sentinel}`] } }), - sessionName, - logPath: path.join(root, 'daemon.log'), - sessionStore, - invoke: async (req) => { - if (req.command === 'press') { - // A real selector miss echoes the RESOLVED (expanded) selector. - return { - ok: false, - error: { - code: 'COMMAND_FAILED', - message: `Selector did not match: ${req.positionals?.[0] ?? ''}`, - hint: `Run find "${sentinel}" for contains matching.`, - }, - }; - } - return { ok: true, data: {} }; - }, - }); - - expect(response.ok).toBe(false); - if (response.ok) return; - // The expanded value appears nowhere in the whole public error. - expect(JSON.stringify(response.error)).not.toContain(sentinel); - // The scrub is a marker replacement, not a drop: the caller still sees - // WHICH variable the selector interpolated. - const divergence = response.error.details?.divergence as { - cause: { message: string; hint?: string }; - }; - expect(divergence.cause.message).toContain(''); - expect(divergence.cause.hint).toContain(''); - expect(response.error.message).toContain(''); -}); - -test('an expanded built-in AD_DEVICE_ID never reaches the public divergence', async () => { - const deviceId = 'BuiltInDeviceId-486b3d4c-8f92-4dc0-b5c6-unique'; - const sessionName = 'static-session-context'; - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-builtin-var-leak-')); - const sessionStore = new SessionStore(path.join(root, 'sessions')); - sessionStore.set(sessionName, makeIosSession(sessionName, { appBundleId: 'com.example.app' })); - const filePath = writeReplayFile(root, ['press label="${AD_DEVICE_ID}"']); - mockDispatchCommand.mockRejectedValue(new Error('no device runner available')); - - const response = await runReplayScriptFile({ - req: baseReq({ positionals: [filePath], flags: { serial: deviceId } }), sessionName, logPath: path.join(root, 'daemon.log'), sessionStore, - invoke: async (req) => ({ - ok: false, - error: { - code: 'COMMAND_FAILED', - message: `Device ${req.positionals?.[0] ?? ''} failed in ${sessionName} context.`, - }, - }), + invoke: async () => ({ ok: true, data: {} }), }); - expect(response.ok).toBe(false); - if (response.ok) return; - expect(JSON.stringify(response.error)).not.toContain(deviceId); - expect(response.error.message).toContain(''); - // AD_SESSION was not expanded, so matching static text remains readable. - expect(response.error.message).toContain(sessionName); + expect(response.ok).toBe(true); }); diff --git a/src/daemon/handlers/__tests__/session-replay-vars.test.ts b/src/daemon/handlers/__tests__/session-replay-vars.test.ts index 037d19a2c..0f3c324f2 100644 --- a/src/daemon/handlers/__tests__/session-replay-vars.test.ts +++ b/src/daemon/handlers/__tests__/session-replay-vars.test.ts @@ -329,44 +329,39 @@ test('readReplayScriptMetadata rejects duplicate env key', () => { ); }); -test('runReplayScriptFile rejects replay -u on scripts with env directives', async () => { - const { response } = await runReplayFixture({ +// ADR 0012 decision 1 / migration step 6: `--update` retirement removed the +// env/interpolation/compat-flow refusal guards below — they existed only to +// protect the now-deleted heal-and-rewrite path, which could not safely +// round-trip `env`/`${VAR}` or serialize Maestro's in-memory flow controls +// back to `.ad`. With no rewrite, `--update` no longer needs to refuse +// anything; it runs exactly like a plain replay. + +test('--update no longer rejects scripts with env directives (the guard existed only for rewrite safety)', async () => { + const { response, calls } = await runReplayFixture({ label: 'env-heal', script: 'context platform=android\nenv APP=settings\nopen ${APP}\n', flags: { replayUpdate: true }, }); - assert.equal(response.ok, false); - if (!response.ok) { - assert.equal(response.error.code, 'INVALID_ARGS'); - assert.match(response.error.message, /replay -u does not yet preserve env directives/); - } + assert.equal(response.ok, true); + assert.deepEqual(calls[0]?.positionals, ['settings']); }); -test('runReplayScriptFile rejects replay -u for Maestro compat flow controls before serialization', async () => { +test('--update no longer rejects Maestro compat flow controls (the guard existed only for rewrite safety)', async () => { const { response } = await runReplayFixture({ label: 'maestro-replay-update-flow-control', script: [ 'appId: demo.app', '---', - '- runFlow:', - ' when:', - ' visible: Feed', - ' commands:', - ' - tapOn: Continue', '- retry:', ' maxRetries: 1', ' commands:', - ' - assertVisible: Feed', + ' - back', '', ].join('\n'), flags: { replayBackend: 'maestro', replayUpdate: true }, }); - assert.equal(response.ok, false); - if (!response.ok) { - assert.equal(response.error.code, 'INVALID_ARGS'); - assert.match(response.error.message, /replay -u is not supported for compat flow input/); - } + assert.equal(response.ok, true); }); test('resolveReplayAction produces dispatch-ready literals for a realistic fixture', () => { diff --git a/src/daemon/handlers/__tests__/session-replay.test.ts b/src/daemon/handlers/__tests__/session-replay.test.ts index fa5544eda..6555c9d9c 100644 --- a/src/daemon/handlers/__tests__/session-replay.test.ts +++ b/src/daemon/handlers/__tests__/session-replay.test.ts @@ -8,7 +8,6 @@ import { LeaseRegistry } from '../../lease-registry.ts'; import type { DaemonRequest, DaemonResponse } from '../../types.ts'; import { makeIosSession } from '../../../__tests__/test-utils/index.ts'; import { buildNestedReplayFlags, handleSessionReplayCommands } from '../session-replay.ts'; -import { collectReplayActionArtifactPaths } from '../session-replay-runtime.ts'; const recordTraceMocks = vi.hoisted(() => ({ handleRecordCommand: vi.fn(), @@ -210,25 +209,6 @@ test('buildNestedReplayFlags strips test-only recordVideo before replay actions assert.deepEqual(result, { platform: 'ios' }); }); -test('collectReplayActionArtifactPaths includes failed action artifact details', () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-artifacts-')); - const snapshotPath = path.join(root, 'failure-snapshot.txt'); - fs.writeFileSync(snapshotPath, 'snapshot'); - - const paths = collectReplayActionArtifactPaths({ - ok: false, - error: { - code: 'COMMAND_FAILED', - message: 'assertion failed', - details: { - artifactPaths: [snapshotPath, path.join(root, 'missing.txt')], - }, - }, - }); - - assert.deepEqual(paths, [snapshotPath]); -}); - test('test --record-video records each replay attempt on the generated test session', async () => { vi.useFakeTimers({ now: 1_000 }); const { root, replayPath, sessionStore, nestedRequests, events } = createRecordVideoFixture(); @@ -289,3 +269,66 @@ test('test --record-video records each replay attempt on the generated test sess false, ); }); + +// --- ADR 0012 decision 4 / migration step 5: `--from` is replay-only --- + +test('test rejects --from with INVALID_ARGS before running the suite', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-test-from-rejected-')); + const replayPath = path.join(root, 'flow.ad'); + fs.writeFileSync(replayPath, 'open "Demo"\nclick "Continue"\n'); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + + const response = await handleSessionReplayCommands({ + req: { + token: 'token', + session: 'default', + command: 'test', + positionals: [replayPath], + flags: { replayFrom: 2, replayPlanDigest: 'deadbeef' }, + meta: { cwd: root }, + }, + sessionName: 'default', + logPath: path.join(root, 'daemon.log'), + sessionStore, + leaseRegistry: new LeaseRegistry(), + invoke: async () => { + throw new Error('test must not start executing when --from is rejected'); + }, + }); + + if (!response) throw new Error('Expected response'); + assert.equal(response.ok, false); + if (response.ok) return; + assert.equal(response.error.code, 'INVALID_ARGS'); + assert.match(response.error.message, /--from/); +}); + +test('test rejects --plan-digest alone with INVALID_ARGS before running the suite', async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-test-digest-rejected-')); + const replayPath = path.join(root, 'flow.ad'); + fs.writeFileSync(replayPath, 'open "Demo"\nclick "Continue"\n'); + const sessionStore = new SessionStore(path.join(root, 'sessions')); + + const response = await handleSessionReplayCommands({ + req: { + token: 'token', + session: 'default', + command: 'test', + positionals: [replayPath], + flags: { replayPlanDigest: 'deadbeef' }, + meta: { cwd: root }, + }, + sessionName: 'default', + logPath: path.join(root, 'daemon.log'), + sessionStore, + leaseRegistry: new LeaseRegistry(), + invoke: async () => { + throw new Error('test must not start executing when --plan-digest is rejected'); + }, + }); + + if (!response) throw new Error('Expected response'); + assert.equal(response.ok, false); + if (response.ok) return; + assert.equal(response.error.code, 'INVALID_ARGS'); +}); diff --git a/src/daemon/handlers/session-replay-divergence.ts b/src/daemon/handlers/session-replay-divergence.ts index 9e2f33201..eb4aab0f3 100644 --- a/src/daemon/handlers/session-replay-divergence.ts +++ b/src/daemon/handlers/session-replay-divergence.ts @@ -16,11 +16,11 @@ import { type Selector, } from '../../selectors/index.ts'; import { collectReplaySelectorCandidates } from './session-replay-heal.ts'; +import { buildReplayDivergenceResume } from './session-replay-resume.ts'; import { formatDivergenceActionLabel, isTouchTargetCommand } from '../../replay/script-utils.ts'; import { SessionStore } from '../session-store.ts'; import type { SessionAction, SessionState } from '../types.ts'; import { - REPLAY_DIVERGENCE_RESUME_NOT_SUPPORTED, REPLAY_DIVERGENCE_SUGGESTION_LIMIT, boundReplayDivergence, createReplayDivergenceSanitizer, @@ -54,6 +54,10 @@ export async function buildReplayFailureDivergence(params: { responseLevel: ResponseLevel | undefined; /** Replay-scope values scrubbed from every divergence string (ADR 0012: expanded variables are never serialized). */ scrubVars?: ReplayVarScrubEntry[]; + /** ADR 0012 migration step 5: the full top-level plan, used to compute `resume.allowed`. */ + planActions: SessionAction[]; + /** SHA-256 digest of the canonical plan `planActions` came from (`computeReplayPlanDigest`). */ + planDigest: string; }): Promise { const { error, @@ -67,6 +71,8 @@ export async function buildReplayFailureDivergence(params: { logPath, responseLevel, scrubVars = [], + planActions, + planDigest, } = params; const sanitize = createReplayDivergenceSanitizer(scrubVars); @@ -107,7 +113,11 @@ export async function buildReplayFailureDivergence(params: { screen, suggestions: suggestions.slice(0, REPLAY_DIVERGENCE_SUGGESTION_LIMIT), suggestionCount: suggestions.length, - resume: REPLAY_DIVERGENCE_RESUME_NOT_SUPPORTED, + resume: buildReplayDivergenceResume({ + failedIndex: index + 1, + actions: planActions, + planDigest, + }), }; return boundReplayDivergence({ diff --git a/src/daemon/handlers/session-replay-heal.ts b/src/daemon/handlers/session-replay-heal.ts index 1c9aef375..bcdd7f5c0 100644 --- a/src/daemon/handlers/session-replay-heal.ts +++ b/src/daemon/handlers/session-replay-heal.ts @@ -1,20 +1,17 @@ -import { dispatchCommand } from '../../core/dispatch.ts'; -import { setSessionSnapshot } from '../session-snapshot.ts'; -import type { RawSnapshotNode, SnapshotBackend, SnapshotState } from '../../kernel/snapshot.ts'; -import { - buildSelectorChainForNode, - resolveSelectorChain, - splitIsSelectorArgs, - splitSelectorFromArgs, - tryParseSelectorChain, -} from '../../selectors/index.ts'; +import { splitIsSelectorArgs, splitSelectorFromArgs } from '../../selectors/index.ts'; import { uniqueStrings } from '../../kernel/collections.ts'; -import { inferFillText } from '../action-utils.ts'; -import type { SessionAction, SessionState } from '../types.ts'; +import type { SessionAction } from '../types.ts'; import { isTouchTargetCommand } from '../../replay/script-utils.ts'; -import { contextFromFlags } from '../context.ts'; -import { SessionStore } from '../session-store.ts'; -import { buildSnapshotState } from './snapshot-capture.ts'; + +/** + * ADR 0012 decision 1 / migration step 6: `--update` retired as an actor — + * this module used to also drive `healReplayAction`'s retry-and-rewrite arm + * (recorded-selector re-resolution feeding a silent `.ad` rewrite). That arm + * is gone; the recorded-selector-candidate extraction below is the one piece + * of the old heal machinery that survives, now read-only, powering the + * ranked `suggestions` list in every divergence report + * (`session-replay-divergence.ts`'s `collectReplayDivergenceSuggestions`). + */ function parseSelectorWaitPositionals(positionals: string[]): { selectorExpression: string | null; @@ -81,99 +78,6 @@ export function collectReplaySelectorCandidates(action: SessionAction): string[] return uniqueStrings(result).filter((entry) => entry.trim().length > 0); } -function collectReplaySelectorChains(action: SessionAction) { - return collectReplaySelectorCandidates(action) - .map((candidate) => tryParseSelectorChain(candidate)) - .filter((chain) => chain !== null); -} - -// fallow-ignore-next-line complexity -export async function healReplayAction(params: { - action: SessionAction; - sessionName: string; - logPath: string; - sessionStore: SessionStore; -}): Promise { - const { action, sessionName, logPath, sessionStore } = params; - if ( - !( - isTouchTargetCommand(action.command) || ['fill', 'get', 'is', 'wait'].includes(action.command) - ) - ) { - return null; - } - - const session = sessionStore.get(sessionName); - if (!session) return null; - const selectorChains = collectReplaySelectorChains(action); - if (selectorChains.length === 0) return null; - - const requiresRect = isTouchTargetCommand(action.command) || action.command === 'fill'; - const allowDisambiguation = - isTouchTargetCommand(action.command) || - action.command === 'fill' || - (action.command === 'get' && action.positionals?.[0] === 'text'); - const snapshot = await captureSnapshotForReplay( - session, - action, - logPath, - requiresRect, - sessionStore, - ); - for (const chain of selectorChains) { - const resolved = resolveSelectorChain(snapshot.nodes, chain, { - platform: session.device.platform, - requireRect: requiresRect, - requireUnique: true, - disambiguateAmbiguous: allowDisambiguation, - }); - if (!resolved) continue; - - const selectorChain = buildSelectorChainForNode(resolved.node, session.device.platform, { - action: - action.command === 'fill' ? 'fill' : isTouchTargetCommand(action.command) ? 'click' : 'get', - }); - const selectorExpression = selectorChain.join(' || '); - - if (isTouchTargetCommand(action.command)) { - return { - ...action, - positionals: - action.command === 'longpress' - ? withLongPressDuration(action, selectorExpression) - : [selectorExpression], - }; - } - if (action.command === 'fill') { - const fillText = inferFillText(action); - if (!fillText) continue; - return { ...action, positionals: [selectorExpression, fillText] }; - } - if (action.command === 'get') { - const sub = action.positionals?.[0]; - if (sub !== 'text' && sub !== 'attrs') continue; - return { ...action, positionals: [sub, selectorExpression] }; - } - if (action.command === 'is') { - const { predicate, split } = splitIsSelectorArgs(action.positionals); - if (!predicate) continue; - const expectedText = split?.rest.join(' ').trim() ?? ''; - const nextPositionals = [predicate, selectorExpression]; - if (predicate === 'text' && expectedText.length > 0) { - nextPositionals.push(expectedText); - } - return { ...action, positionals: nextPositionals }; - } - if (action.command === 'wait') { - const { selectorTimeout } = parseSelectorWaitPositionals(action.positionals ?? []); - const nextPositionals = [selectorExpression]; - if (selectorTimeout) nextPositionals.push(selectorTimeout); - return { ...action, positionals: nextPositionals }; - } - } - return null; -} - function readTargetSelectorPositionals(action: SessionAction): string[] { const positionals = action.positionals ?? []; if (action.command !== 'longpress') return positionals; @@ -183,51 +87,7 @@ function readTargetSelectorPositionals(action: SessionAction): string[] { : positionals; } -function withLongPressDuration(action: SessionAction, selectorExpression: string): string[] { - const durationMs = - typeof action.result?.durationMs === 'number' - ? String(action.result.durationMs) - : readLongPressDurationFromPositionals(action.positionals ?? []); - return durationMs ? [selectorExpression, durationMs] : [selectorExpression]; -} - -function readLongPressDurationFromPositionals(positionals: string[]): string | undefined { - const last = positionals.at(-1); - return positionals.length > 1 && isFiniteNumberString(last) ? last : undefined; -} - function isFiniteNumberString(value: string | undefined): boolean { if (value === undefined || value.trim() === '') return false; return Number.isFinite(Number(value)); } - -async function captureSnapshotForReplay( - session: SessionState, - action: SessionAction, - logPath: string, - interactiveOnly: boolean, - sessionStore: SessionStore, -): Promise { - const data = (await dispatchCommand(session.device, 'snapshot', [], action.flags?.out, { - ...contextFromFlags( - logPath, - { - ...(action.flags ?? {}), - snapshotInteractiveOnly: interactiveOnly, - }, - session.appBundleId, - session.trace?.outPath, - ), - })) as { - nodes?: RawSnapshotNode[]; - truncated?: boolean; - backend?: SnapshotBackend; - }; - const snapshot = buildSnapshotState(data, { - ...(action.flags ?? {}), - snapshotInteractiveOnly: interactiveOnly, - }); - setSessionSnapshot(session, snapshot); - sessionStore.set(session.name, session); - return snapshot; -} diff --git a/src/daemon/handlers/session-replay-resume.ts b/src/daemon/handlers/session-replay-resume.ts new file mode 100644 index 000000000..e88ccfb1e --- /dev/null +++ b/src/daemon/handlers/session-replay-resume.ts @@ -0,0 +1,74 @@ +import { MAESTRO_RUNTIME_COMMAND } from '../../compat/maestro/runtime-commands.ts'; +import type { SessionAction } from '../types.ts'; +import type { ReplayDivergenceResume } from '../../replay/divergence.ts'; + +/** + * ADR 0012 decision 4 / migration step 5: resume does not reconstruct + * execution state. For a target `from` (1-based, into the SAME top-level + * `actions` array the replay runtime iterates) greater than 1, preflight + * rejects when any SKIPPED step (`1..from-1`) can produce `outputEnv` + * values a later step might consume, or when the skipped range or the + * resume target itself is a runtime control-flow wrapper (`retry` or + * `maestroRunFlowWhen` — evaluated dynamically, never expanded into + * individually addressable plan indices, so skipping into/over one cannot be + * proven safe). `from === 1` skips nothing and is always allowed. + */ +export type ReplayResumePreflightResult = { allowed: true } | { allowed: false; reason: string }; + +// The only outputEnv producer today (`invokeMaestroRunScript`, +// `compat/maestro/runtime.ts`); a plain command's response is never merged +// into replay var scope (`readReplayOutputEnv`, +// `session-replay-action-runtime.ts`). +const OUTPUT_ENV_PRODUCING_COMMAND: string = MAESTRO_RUNTIME_COMMAND.runScript; + +export function evaluateReplayResumePreflight(params: { + from: number; + actions: SessionAction[]; +}): ReplayResumePreflightResult { + const { from, actions } = params; + if (from <= 1) return { allowed: true }; + + for (let index = 0; index <= from - 2; index += 1) { + const action = actions[index]; + if (!action) continue; + if (action.replayControl) { + return { + allowed: false, + reason: `step ${index + 1} is inside runtime control flow (${action.replayControl.kind}); skipping it without executing it cannot be proven safe.`, + }; + } + if (producesOutputEnv(action)) { + return { + allowed: false, + reason: `step ${index + 1} (${action.command}) can produce outputEnv values a later step may consume; skipping it without executing it cannot be proven safe.`, + }; + } + } + + const target = actions[from - 1]; + if (target?.replayControl) { + return { + allowed: false, + reason: `step ${from} is inside runtime control flow (${target.replayControl.kind}); it cannot be safely resumed into.`, + }; + } + + return { allowed: true }; +} + +function producesOutputEnv(action: SessionAction): boolean { + return action.command === OUTPUT_ENV_PRODUCING_COMMAND; +} + +/** Builds the `resume` object attached to every divergence report. */ +export function buildReplayDivergenceResume(params: { + failedIndex: number; // 1-based + actions: SessionAction[]; + planDigest: string; +}): ReplayDivergenceResume { + const { failedIndex, actions, planDigest } = params; + const preflight = evaluateReplayResumePreflight({ from: failedIndex, actions }); + return preflight.allowed + ? { allowed: true, from: failedIndex, planDigest } + : { allowed: false, from: failedIndex, planDigest, reason: preflight.reason }; +} diff --git a/src/daemon/handlers/session-replay-runtime-artifacts.ts b/src/daemon/handlers/session-replay-runtime-artifacts.ts new file mode 100644 index 000000000..4995c78d1 --- /dev/null +++ b/src/daemon/handlers/session-replay-runtime-artifacts.ts @@ -0,0 +1,55 @@ +import fs from 'node:fs'; +import type { DaemonResponse } from '../types.ts'; + +export function collectReplayActionArtifactPaths(response: DaemonResponse): string[] { + const candidates = response.ok + ? collectSuccessfulArtifactCandidates(response.data) + : collectFailureArtifactCandidates(response.error.details?.artifactPaths); + return uniqueExistingArtifactPaths(candidates); +} + +type ReplayResponseData = Extract['data']; + +function collectFailureArtifactCandidates(value: unknown): string[] { + return Array.isArray(value) ? value.filter(isString) : []; +} + +function collectSuccessfulArtifactCandidates(data: ReplayResponseData): string[] { + if (!data) return []; + return [ + readString(data.path), + readString(data.outPath), + ...collectNestedArtifactCandidates(data.artifacts), + ].filter(isString); +} + +function collectNestedArtifactCandidates(value: unknown): string[] { + if (!Array.isArray(value)) return []; + return value.map(readNestedArtifactPath).filter(isString); +} + +function readNestedArtifactPath(value: unknown): string | undefined { + if (!value || typeof value !== 'object' || Array.isArray(value)) return undefined; + const artifact = value as Record; + return readString(artifact.localPath) ?? readString(artifact.path); +} + +function readString(value: unknown): string | undefined { + return typeof value === 'string' ? value : undefined; +} + +function isString(value: unknown): value is string { + return typeof value === 'string'; +} + +function uniqueExistingArtifactPaths(candidates: string[]): string[] { + return [...new Set(candidates.filter(isReplayArtifactPath))]; +} + +function isReplayArtifactPath(candidate: string): boolean { + try { + return fs.statSync(candidate).isFile(); + } catch { + return false; + } +} diff --git a/src/daemon/handlers/session-replay-runtime-failure-response.ts b/src/daemon/handlers/session-replay-runtime-failure-response.ts new file mode 100644 index 000000000..e45f23f99 --- /dev/null +++ b/src/daemon/handlers/session-replay-runtime-failure-response.ts @@ -0,0 +1,86 @@ +import { scrubReplayVarValues, type ReplayVarScrubEntry } from '../../replay/divergence.ts'; +import { formatDivergenceActionLabel } from '../../replay/script-utils.ts'; +import type { SnapshotDiagnosticsSummary } from '../../snapshot-diagnostics.ts'; +import { buildDisplayPositionals } from '../session-event-action.ts'; +import type { DaemonResponse, SessionAction } from '../types.ts'; + +export type ReplayFailureCause = Extract['error']; + +export function hoistReplayFailureCauseDiagnosticMeta( + error: ReplayFailureCause, +): ReplayFailureCause { + return { + ...error, + hint: error.hint ?? readStringDetail(error.details, 'hint'), + diagnosticId: error.diagnosticId ?? readStringDetail(error.details, 'diagnosticId'), + logPath: error.logPath ?? readStringDetail(error.details, 'logPath'), + }; +} + +export function buildReplayDivergenceFailureResponse(params: { + error: ReplayFailureCause; + action: SessionAction; + step: number; + replayPath: string; + artifactPaths: string[]; + snapshotDiagnostics?: SnapshotDiagnosticsSummary; + divergence: unknown; + scrubVars: ReplayVarScrubEntry[]; +}): DaemonResponse { + const { + error, + action, + step, + replayPath, + artifactPaths, + snapshotDiagnostics, + divergence, + scrubVars, + } = params; + return { + ok: false, + error: { + code: 'REPLAY_DIVERGENCE', + message: scrubReplayVarValues( + `Replay failed at step ${step} (${formatDivergenceActionLabel(action)}): ${error.message}`, + scrubVars, + ), + hint: error.hint === undefined ? undefined : scrubReplayVarValues(error.hint, scrubVars), + diagnosticId: error.diagnosticId, + logPath: error.logPath, + ...(error.retriable !== undefined ? { retriable: error.retriable } : {}), + ...(error.supportedOn !== undefined ? { supportedOn: error.supportedOn } : {}), + details: { + ...pickSafeCauseDetails(error.details), + replayPath, + step, + action: action.command, + positionals: buildDisplayPositionals(action) ?? [], + artifactPaths, + ...(snapshotDiagnostics ? { snapshotDiagnostics } : {}), + divergence, + }, + }, + }; +} + +function readStringDetail( + details: Record | undefined, + key: string, +): string | undefined { + const value = details?.[key]; + return typeof value === 'string' && value.length > 0 ? value : undefined; +} + +const SAFE_CAUSE_DETAIL_KEYS = ['reason', 'retriable', 'supportedOn'] as const; + +function pickSafeCauseDetails( + details: Record | undefined, +): Record { + if (!details) return {}; + const safe: Record = {}; + for (const key of SAFE_CAUSE_DETAIL_KEYS) { + if (details[key] !== undefined) safe[key] = details[key]; + } + return safe; +} diff --git a/src/daemon/handlers/session-replay-runtime-failure.ts b/src/daemon/handlers/session-replay-runtime-failure.ts new file mode 100644 index 000000000..eca5d9cc0 --- /dev/null +++ b/src/daemon/handlers/session-replay-runtime-failure.ts @@ -0,0 +1,110 @@ +import { collectReplayScrubbableVarValues, type ReplayVarScope } from '../../replay/vars.ts'; +import { + summarizeSnapshotTimingSamples, + type SnapshotDiagnosticsSummary, + type SnapshotTimingSample, +} from '../../snapshot-diagnostics.ts'; +import { SessionStore } from '../session-store.ts'; +import type { DaemonRequest, DaemonResponse, SessionAction } from '../types.ts'; +import { buildReplayFailureDivergence } from './session-replay-divergence.ts'; +import { + buildReplayDivergenceFailureResponse, + hoistReplayFailureCauseDiagnosticMeta, +} from './session-replay-runtime-failure-response.ts'; + +export async function withReplayFailureDiagnostics(params: { + response: DaemonResponse; + action: SessionAction; + index: number; + replayPath: string; + sourcePath: string; + sourceLine: number; + artifactPaths: string[]; + snapshotDiagnosticSamples: SnapshotTimingSample[]; + scope: ReplayVarScope; + req: DaemonRequest; + sessionName: string; + sessionStore: SessionStore; + logPath: string; + planActions: SessionAction[]; + planDigest: string; +}): Promise { + return await withReplayFailureContext({ + ...params, + snapshotDiagnostics: summarizeSnapshotTimingSamples(params.snapshotDiagnosticSamples), + }); +} + +async function withReplayFailureContext(params: { + response: DaemonResponse; + action: SessionAction; + index: number; + replayPath: string; + sourcePath: string; + sourceLine: number; + artifactPaths?: string[]; + snapshotDiagnostics?: SnapshotDiagnosticsSummary; + scope: ReplayVarScope; + req: DaemonRequest; + sessionName: string; + sessionStore: SessionStore; + logPath: string; + planActions: SessionAction[]; + planDigest: string; +}): Promise { + const { + response, + action, + index, + replayPath, + sourcePath, + sourceLine, + artifactPaths = [], + snapshotDiagnostics, + scope, + req, + sessionName, + sessionStore, + logPath, + planActions, + planDigest, + } = params; + if (response.ok) return response; + const failureSource = readReplayFailureSource(response.error.details?.replaySource); + const scrubVars = collectReplayScrubbableVarValues(scope); + const cause = hoistReplayFailureCauseDiagnosticMeta(response.error); + const divergence = await buildReplayFailureDivergence({ + error: cause, + action, + index, + sourcePath: failureSource?.path ?? sourcePath, + sourceLine: failureSource?.line ?? sourceLine, + session: sessionStore.get(sessionName), + sessionName, + sessionStore, + logPath, + responseLevel: req.meta?.responseLevel, + scrubVars, + planActions, + planDigest, + }); + return buildReplayDivergenceFailureResponse({ + error: cause, + action, + step: index + 1, + replayPath, + artifactPaths, + snapshotDiagnostics, + divergence, + scrubVars, + }); +} + +function readReplayFailureSource(value: unknown): { path?: string; line?: number } | undefined { + if (!value || typeof value !== 'object' || Array.isArray(value)) return undefined; + const record = value as Record; + const path = typeof record.path === 'string' && record.path.length > 0 ? record.path : undefined; + const line = typeof record.line === 'number' ? record.line : undefined; + if (path === undefined && line === undefined) return undefined; + return { path, line }; +} diff --git a/src/daemon/handlers/session-replay-runtime-plan.ts b/src/daemon/handlers/session-replay-runtime-plan.ts new file mode 100644 index 000000000..81fe9f121 --- /dev/null +++ b/src/daemon/handlers/session-replay-runtime-plan.ts @@ -0,0 +1,81 @@ +import type { CommandFlags } from '../../core/dispatch.ts'; +import type { ReplayPlanDigestMetadata } from '../../replay/plan-digest.ts'; +import type { ReplayScriptMetadata } from '../../replay/script.ts'; +import type { DaemonResponse, SessionAction } from '../types.ts'; +import { evaluateReplayResumePreflight } from './session-replay-resume.ts'; +import { errorResponse } from './response.ts'; + +export function buildReplayMetadataFlags( + flags: CommandFlags | undefined, + metadata: ReplayScriptMetadata, +): CommandFlags { + return { + ...(flags ?? {}), + ...(metadata.platform !== undefined && flags?.platform === undefined + ? { platform: metadata.platform } + : {}), + ...(metadata.target !== undefined && flags?.target === undefined + ? { target: metadata.target } + : {}), + }; +} + +/** The digest binds the same platform/target values the replay invokes with. */ +export function readEffectiveReplayPlanDigestMetadata( + flags: CommandFlags | undefined, +): ReplayPlanDigestMetadata { + return { + platform: typeof flags?.platform === 'string' ? flags.platform : undefined, + target: typeof flags?.target === 'string' ? flags.target : undefined, + }; +} + +type ReplayEntryIndexResult = { ok: true; value: number } | { ok: false; response: DaemonResponse }; + +/** + * Resolves `--from`/`--plan-digest` into a 0-based loop entry index before + * any device action. `--from` is 1-based and matches divergence step indices. + */ +export function resolveReplayEntryIndex( + flags: CommandFlags | undefined, + actionCount: number, + planDigest: string, + actions: SessionAction[], +): ReplayEntryIndexResult { + const from = flags?.replayFrom; + const digest = flags?.replayPlanDigest; + if (from === undefined && digest === undefined) return { ok: true, value: 0 }; + if (from === undefined || digest === undefined) { + return invalidReplayEntryIndex( + 'replay --from requires --plan-digest (and --plan-digest requires --from).', + ); + } + const message = validateReplayResumeRequest({ from, digest, planDigest, actionCount, actions }); + return message ? invalidReplayEntryIndex(message) : { ok: true, value: from - 1 }; +} + +function invalidReplayEntryIndex(message: string): ReplayEntryIndexResult { + return { ok: false, response: errorResponse('INVALID_ARGS', message) }; +} + +function validateReplayResumeRequest(params: { + from: number; + digest: string; + planDigest: string; + actionCount: number; + actions: SessionAction[]; +}): string | undefined { + const { from, digest, planDigest, actionCount, actions } = params; + if (!Number.isInteger(from) || from < 1 || from > actionCount) { + return `replay --from ${from} is out of range for a ${actionCount}-step plan.`; + } + if (digest !== planDigest) { + return ( + 'replay --plan-digest does not match the current plan digest; the script, its includes, or its ' + + 'platform-conditioned expansion changed since the divergence report was generated. Run a fresh full ' + + 'replay to get a new digest.' + ); + } + const preflight = evaluateReplayResumePreflight({ from, actions }); + return preflight.allowed ? undefined : `replay --from ${from} cannot resume: ${preflight.reason}`; +} diff --git a/src/daemon/handlers/session-replay-runtime.ts b/src/daemon/handlers/session-replay-runtime.ts index 0ce752f5d..b7c2276f2 100644 --- a/src/daemon/handlers/session-replay-runtime.ts +++ b/src/daemon/handlers/session-replay-runtime.ts @@ -1,6 +1,5 @@ import fs from 'node:fs'; import path from 'node:path'; -import { type CommandFlags } from '../../core/dispatch.ts'; import { parseReplayInput } from '../../compat/replay-input.ts'; import { asAppError } from '../../kernel/errors.ts'; import type { DaemonInvokeFn, DaemonRequest, DaemonResponse, SessionAction } from '../types.ts'; @@ -10,30 +9,30 @@ import { type ReplayTestProgressEvent, } from '../../request/progress.ts'; import { SessionStore } from '../session-store.ts'; -import { type ReplayScriptMetadata, writeReplayScript } from '../../replay/script.ts'; -import { healReplayAction } from './session-replay-heal.ts'; -import { formatDivergenceActionLabel } from '../../replay/script-utils.ts'; -import { buildDisplayPositionals } from '../session-event-action.ts'; +import { type ReplayScriptMetadata } from '../../replay/script.ts'; +import { computeReplayPlanDigest } from '../../replay/plan-digest.ts'; import { errorResponse } from './response.ts'; import { invokeReplayAction } from './session-replay-action-runtime.ts'; import { tryParseSelectorChain } from '../../selectors/index.ts'; import { buildReplayVarScope, - collectReplayScrubbableVarValues, collectReplayShellEnv, parseReplayCliEnvEntries, readReplayCliEnvEntries, readReplayShellEnvSource, - type ReplayVarScope, } from '../../replay/vars.ts'; import { summarizeSnapshotTimingSamples, - type SnapshotDiagnosticsSummary, type SnapshotTimingSample, } from '../../snapshot-diagnostics.ts'; -import { buildReplayFailureDivergence } from './session-replay-divergence.ts'; -import { scrubReplayVarValues, type ReplayVarScrubEntry } from '../../replay/divergence.ts'; import type { ReplayCommandResult } from '../../contracts/replay.ts'; +import { collectReplayActionArtifactPaths } from './session-replay-runtime-artifacts.ts'; +import { withReplayFailureDiagnostics } from './session-replay-runtime-failure.ts'; +import { + buildReplayMetadataFlags, + readEffectiveReplayPlanDigestMetadata, + resolveReplayEntryIndex, +} from './session-replay-runtime-plan.ts'; // fallow-ignore-next-line complexity export async function runReplayScriptFile(params: { @@ -73,21 +72,17 @@ export async function runReplayScriptFile(params: { const actions = parsed.actions; const actionLines = parsed.actionLines; const actionSourcePaths = parsed.actionSourcePaths; - if (req.flags?.replayUpdate === true && parsed.updateUnsupportedMessage) { - return errorResponse('INVALID_ARGS', parsed.updateUnsupportedMessage); - } - if (req.flags?.replayUpdate === true && metadata.env && Object.keys(metadata.env).length > 0) { - return errorResponse( - 'INVALID_ARGS', - 'replay -u does not yet preserve env directives. Temporarily remove the env lines, run replay -u, then restore them.', - ); - } - if (req.flags?.replayUpdate === true && actionsContainInterpolation(actions)) { - return errorResponse( - 'INVALID_ARGS', - 'replay -u does not yet preserve ${VAR} substitutions. Resolve or inline the variables before running with -u.', - ); - } + const planDigest = computeReplayPlanDigest({ + actions, + actionLines, + actionSourcePaths, + metadata: readEffectiveReplayPlanDigestMetadata(replayReq.flags), + }); + // ADR 0012 decision 4 / migration step 5: resume preflight, entirely + // before any device action. `test` never reaches here with either flag + // set (rejected earlier, in handleSessionReplayCommands). + const entryIndex = resolveReplayEntryIndex(req.flags, actions.length, planDigest, actions); + if (!entryIndex.ok) return entryIndex.response; const scope = buildReplayVarScope({ builtins: buildReplayBuiltinVars({ req: replayReq, @@ -99,7 +94,6 @@ export async function runReplayScriptFile(params: { shellEnv: collectReplayShellEnv(readReplayShellEnvSource(req.flags?.replayShellEnv)), cliEnv: parseReplayCliEnvEntries(readReplayCliEnvEntries(req.flags?.replayEnv)), }); - const shouldUpdate = req.flags?.replayUpdate === true; const actionTracePath = tracePath ?? sessionStore.get(sessionName)?.trace?.outPath; const snapshotDiagnosticSamples: SnapshotTimingSample[] = []; const failStep = (failedResponse: DaemonResponse, failedAction: SessionAction, index: number) => @@ -117,15 +111,16 @@ export async function runReplayScriptFile(params: { sessionName, sessionStore, logPath, + planActions: actions, + planDigest, }); - let healed = 0; - for (let index = 0; index < actions.length; index += 1) { + for (let index = entryIndex.value; index < actions.length; index += 1) { const action = actions[index]; if (!action || action.command === 'replay') continue; emitReplayTestActionProgress(resolved, index, actions.length, action); const sampleStart = readSessionSnapshotSampleCount(sessionStore, sessionName); - let response = await invokeReplayAction({ + const response = await invokeReplayAction({ req: replayReq, sessionName, action, @@ -140,65 +135,28 @@ export async function runReplayScriptFile(params: { snapshotDiagnosticSamples.push( ...readSessionSnapshotSamplesSince(sessionStore, sessionName, sampleStart), ); - if (response.ok) { - collectReplayActionArtifactPaths(response).forEach((entry) => artifactPaths.add(entry)); - continue; - } collectReplayActionArtifactPaths(response).forEach((entry) => artifactPaths.add(entry)); - if (!shouldUpdate) { - return await failStep(response, action, index); - } - - const nextAction = await healReplayAction({ - action, - sessionName, - logPath, - sessionStore, - }); - if (!nextAction) { - return await failStep(response, action, index); - } - - actions[index] = nextAction; - const healedSampleStart = readSessionSnapshotSampleCount(sessionStore, sessionName); - response = await invokeReplayAction({ - req: replayReq, - sessionName, - action: nextAction, - scope, - filePath: resolved, - line: actionLines[index] ?? 1, - sourcePath: actionSourcePaths?.[index], - step: index + 1, - tracePath: actionTracePath, - invoke, - }); - snapshotDiagnosticSamples.push( - ...readSessionSnapshotSamplesSince(sessionStore, sessionName, healedSampleStart), - ); if (!response.ok) { - collectReplayActionArtifactPaths(response).forEach((entry) => artifactPaths.add(entry)); - return await failStep(response, nextAction, index); + return await failStep(response, action, index); } - collectReplayActionArtifactPaths(response).forEach((entry) => artifactPaths.add(entry)); - healed += 1; } - if (shouldUpdate && healed > 0) { - writeReplayScript(resolved, actions, sessionStore.get(sessionName)); - } + const replayedCount = actions.length - entryIndex.value; const snapshotDiagnosticsSummary = summarizeSnapshotTimingSamples(snapshotDiagnosticSamples); const wallClockMs = Date.now() - startedAt; return { ok: true, data: { - replayed: actions.length, - healed, + replayed: replayedCount, + // ADR 0012 migration step 6: `--update` retired as an actor; it never + // healed anything in this run, so the count is always 0. Kept on the + // wire shape for existing reporters/consumers (test summary, JUnit). + healed: 0, session: sessionName, artifactPaths: [...artifactPaths], ...(snapshotDiagnosticsSummary ? { snapshotDiagnostics: snapshotDiagnosticsSummary } : {}), // ADR 0012: one-line text success summary; --json shape is additive. - message: formatReplaySuccessMessage(actions.length, wallClockMs), + message: formatReplaySuccessMessage(replayedCount, wallClockMs), } satisfies ReplayCommandResult, }; } catch (err) { @@ -312,251 +270,12 @@ function readSelectorDisplayValue(selector: string | undefined): string | undefi return first && values.every((value) => value === first) ? first : undefined; } -function buildReplayMetadataFlags( - flags: CommandFlags | undefined, - metadata: ReplayScriptMetadata, -): CommandFlags { - return { - ...(flags ?? {}), - ...(metadata.platform !== undefined && flags?.platform === undefined - ? { platform: metadata.platform } - : {}), - ...(metadata.target !== undefined && flags?.target === undefined - ? { target: metadata.target } - : {}), - }; -} - -async function withReplayFailureDiagnostics(params: { - response: DaemonResponse; - action: SessionAction; - index: number; - replayPath: string; - sourcePath: string; - sourceLine: number; - artifactPaths: string[]; - snapshotDiagnosticSamples: SnapshotTimingSample[]; - scope: ReplayVarScope; - req: DaemonRequest; - sessionName: string; - sessionStore: SessionStore; - logPath: string; -}): Promise { - return await withReplayFailureContext({ - ...params, - snapshotDiagnostics: summarizeSnapshotTimingSamples(params.snapshotDiagnosticSamples), - }); -} - -/** - * Single choke point for replay step failures (ADR 0012 migration step 2): - * returns `REPLAY_DIVERGENCE` with a bounded `details.divergence` report; - * the original code/message/hint move into `divergence.cause` verbatim, and - * the pre-existing flat detail fields are kept for their consumers. - */ -async function withReplayFailureContext(params: { - response: DaemonResponse; - action: SessionAction; - index: number; - replayPath: string; - sourcePath: string; - sourceLine: number; - artifactPaths?: string[]; - snapshotDiagnostics?: SnapshotDiagnosticsSummary; - scope: ReplayVarScope; - req: DaemonRequest; - sessionName: string; - sessionStore: SessionStore; - logPath: string; -}): Promise { - const { - response, - action, - index, - replayPath, - sourcePath, - sourceLine, - artifactPaths = [], - snapshotDiagnostics, - scope, - req, - sessionName, - sessionStore, - logPath, - } = params; - if (response.ok) return response; - // The failing action's own source (attached by withReplayFailureSource, - // deepest failure wins) beats the top-level wrapper's source. - const failureSource = readReplayFailureSource(response.error.details?.replaySource); - // Computed at failure time so runtime outputEnv merges are included. - const scrubVars = collectReplayScrubbableVarValues(scope); - const cause = hoistCauseDiagnosticMeta(response.error); - const divergence = await buildReplayFailureDivergence({ - error: cause, - action, - index, - sourcePath: failureSource?.path ?? sourcePath, - sourceLine: failureSource?.line ?? sourceLine, - session: sessionStore.get(sessionName), - sessionName, - sessionStore, - logPath, - responseLevel: req.meta?.responseLevel, - scrubVars, - }); - return buildReplayDivergenceFailureResponse({ - error: cause, - action, - step: index + 1, - replayPath, - artifactPaths, - snapshotDiagnostics, - divergence, - scrubVars, - }); -} - -type ReplayFailureCause = Extract['error']; - -// Throw sites may carry hint/diagnosticId/logPath inside details (the -// documented AppErrorDetails meta keys, normally lifted by normalizeError); -// the categorical cause-detail strip below would lose them, so hoist onto the -// error fields first. -function hoistCauseDiagnosticMeta(error: ReplayFailureCause): ReplayFailureCause { - return { - ...error, - hint: error.hint ?? readStringDetail(error.details, 'hint'), - diagnosticId: error.diagnosticId ?? readStringDetail(error.details, 'diagnosticId'), - logPath: error.logPath ?? readStringDetail(error.details, 'logPath'), - }; -} - -function readStringDetail( - details: Record | undefined, - key: string, -): string | undefined { - const value = details?.[key]; - return typeof value === 'string' && value.length > 0 ? value : undefined; -} - -// ADR 0012: arbitrary nested cause details are never serialized into the -// public divergence error — value-bearing command details (fill -// verification's `expected`/`actual`, selector diagnostics, process output) -// are categorically dropped; only machine-dispatchable signals survive. -const SAFE_CAUSE_DETAIL_KEYS = ['reason', 'retriable', 'supportedOn'] as const; - -function pickSafeCauseDetails( - details: Record | undefined, -): Record { - if (!details) return {}; - const safe: Record = {}; - for (const key of SAFE_CAUSE_DETAIL_KEYS) { - if (details[key] !== undefined) safe[key] = details[key]; - } - return safe; -} - -/** Pure wire shaping for the REPLAY_DIVERGENCE failure response. */ -function buildReplayDivergenceFailureResponse(params: { - error: Extract['error']; - action: SessionAction; - step: number; - replayPath: string; - artifactPaths: string[]; - snapshotDiagnostics?: SnapshotDiagnosticsSummary; - divergence: unknown; - scrubVars: ReplayVarScrubEntry[]; -}): DaemonResponse { - const { - error, - action, - step, - replayPath, - artifactPaths, - snapshotDiagnostics, - divergence, - scrubVars, - } = params; - return { - ok: false, - error: { - code: 'REPLAY_DIVERGENCE', - // The cause message can echo an expanded selector; the top-level - // message gets the same categorical variable scrub as the report. - message: scrubReplayVarValues( - `Replay failed at step ${step} (${formatDivergenceActionLabel(action)}): ${error.message}`, - scrubVars, - ), - hint: error.hint === undefined ? undefined : scrubReplayVarValues(error.hint, scrubVars), - diagnosticId: error.diagnosticId, - logPath: error.logPath, - ...(error.retriable !== undefined ? { retriable: error.retriable } : {}), - ...(error.supportedOn !== undefined ? { supportedOn: error.supportedOn } : {}), - details: { - ...pickSafeCauseDetails(error.details), - replayPath, - step, - action: action.command, - // Categorical text hiding (``), never raw fill/type/ - // payload text — the same event-log sanitizer. - positionals: buildDisplayPositionals(action) ?? [], - artifactPaths, - ...(snapshotDiagnostics ? { snapshotDiagnostics } : {}), - divergence, - }, - }, - }; -} - -function readReplayFailureSource(value: unknown): { path?: string; line?: number } | undefined { - if (!value || typeof value !== 'object' || Array.isArray(value)) return undefined; - const record = value as Record; - const path = typeof record.path === 'string' && record.path.length > 0 ? record.path : undefined; - const line = typeof record.line === 'number' ? record.line : undefined; - if (path === undefined && line === undefined) return undefined; - return { path, line }; -} - function formatReplaySuccessMessage(replayed: number, wallClockMs: number): string { const seconds = (wallClockMs / 1000).toFixed(1); const noun = replayed === 1 ? 'step' : 'steps'; return `Replayed ${replayed} ${noun} in ${seconds}s`; } -// fallow-ignore-next-line complexity -export function collectReplayActionArtifactPaths(response: DaemonResponse): string[] { - if (!response.ok) { - const paths = response.error.details?.artifactPaths; - return Array.isArray(paths) - ? [ - ...new Set( - paths.filter( - (candidate): candidate is string => - typeof candidate === 'string' && isReplayArtifactPath(candidate), - ), - ), - ] - : []; - } - if (!response.data) return []; - const candidates: string[] = []; - if (typeof response.data.path === 'string') candidates.push(response.data.path); - if (typeof response.data.outPath === 'string') candidates.push(response.data.outPath); - if (Array.isArray(response.data.artifacts)) { - for (const artifact of response.data.artifacts) { - if (!artifact || typeof artifact !== 'object') continue; - const artifactRecord = artifact as Record; - const localPath = - typeof artifactRecord.localPath === 'string' ? artifactRecord.localPath : undefined; - const artifactPath = - typeof artifactRecord.path === 'string' ? artifactRecord.path : undefined; - if (localPath) candidates.push(localPath); - else if (artifactPath) candidates.push(artifactPath); - } - } - return [...new Set(candidates.filter((candidate) => isReplayArtifactPath(candidate)))]; -} - function readSessionSnapshotSampleCount(sessionStore: SessionStore, sessionName: string): number { return sessionStore.get(sessionName)?.snapshotDiagnostics?.samples.length ?? 0; } @@ -568,30 +287,3 @@ function readSessionSnapshotSamplesSince( ): SnapshotTimingSample[] { return sessionStore.get(sessionName)?.snapshotDiagnostics?.samples.slice(start) ?? []; } - -function isReplayArtifactPath(candidate: string): boolean { - try { - return fs.statSync(candidate).isFile(); - } catch { - return false; - } -} - -// fallow-ignore-next-line complexity -function actionsContainInterpolation(actions: SessionAction[]): boolean { - for (const action of actions) { - for (const positional of action.positionals ?? []) { - if (typeof positional === 'string' && positional.includes('${')) return true; - } - if (containsInterpolation(action.flags)) return true; - if (containsInterpolation(action.runtime)) return true; - } - return false; -} - -function containsInterpolation(value: unknown): boolean { - if (typeof value === 'string') return value.includes('${'); - if (Array.isArray(value)) return value.some(containsInterpolation); - if (value && typeof value === 'object') return Object.values(value).some(containsInterpolation); - return false; -} diff --git a/src/daemon/handlers/session-replay-video-recording.ts b/src/daemon/handlers/session-replay-video-recording.ts index ff2b18597..a64904e5c 100644 --- a/src/daemon/handlers/session-replay-video-recording.ts +++ b/src/daemon/handlers/session-replay-video-recording.ts @@ -5,7 +5,7 @@ import { emitDiagnostic } from '../../utils/diagnostics.ts'; import { sleep } from '../../utils/timeouts.ts'; import { handleRecordCommand } from './record-trace-recording.ts'; import { appendReplayTestTimingEvent } from './session-test-runtime.ts'; -import { collectReplayActionArtifactPaths } from './session-replay-runtime.ts'; +import { collectReplayActionArtifactPaths } from './session-replay-runtime-artifacts.ts'; import { defaultRecordingPath, recordingExtensionForPlatform, diff --git a/src/daemon/handlers/session-replay.ts b/src/daemon/handlers/session-replay.ts index e3e9df950..9bd82ab76 100644 --- a/src/daemon/handlers/session-replay.ts +++ b/src/daemon/handlers/session-replay.ts @@ -3,7 +3,9 @@ import type { DaemonInvokeFn, DaemonRequest, DaemonResponse } from '../types.ts' import { SessionStore } from '../session-store.ts'; import { runReplayTestSuite } from './session-test.ts'; import { handleCloseCommand } from './session-close.ts'; -import { collectReplayActionArtifactPaths, runReplayScriptFile } from './session-replay-runtime.ts'; +import { runReplayScriptFile } from './session-replay-runtime.ts'; +import { collectReplayActionArtifactPaths } from './session-replay-runtime-artifacts.ts'; +import { errorResponse } from './response.ts'; import type { ReplayScriptMetadata } from '../../replay/script.ts'; import { buildReplayTestShardFlags, type ReplayTestShardContext } from './session-test-sharding.ts'; import type { LeaseRegistry } from '../lease-registry.ts'; @@ -69,6 +71,16 @@ export async function handleSessionReplayCommands(params: { } if (req.command === 'test') { + // ADR 0012 decision 4 / migration step 5: `--from` is replay-only. `test` + // shares replay execution (below, via a nested `command: 'replay'` + // request per matched file) but must remain a full, deterministic suite + // run, so this is the one place that still knows the ORIGINAL command. + if (req.flags?.replayFrom !== undefined || req.flags?.replayPlanDigest !== undefined) { + return errorResponse( + 'INVALID_ARGS', + 'test does not support --from/--plan-digest; resume is replay-only. Run the failing script directly with replay --from.', + ); + } return await runReplayTestSuite({ req, sessionName, diff --git a/src/daemon/session-snapshot.ts b/src/daemon/session-snapshot.ts index 6128f5b92..b2b43c08f 100644 --- a/src/daemon/session-snapshot.ts +++ b/src/daemon/session-snapshot.ts @@ -26,7 +26,6 @@ export const STALE_SNAPSHOT_REFS_WARNING = * --verify evidence captures routed through the interaction runtime * - handlers/interaction-snapshot.ts — Android ref-freshness refreshes and * recording reference-frame captures - * - handlers/session-replay-heal.ts — replay heal re-captures * - request-generic-dispatch.ts — screenshot --overlay-refs capture (the * overlay burns in at most a scored subset of refs, so it does NOT count as * issuing the full ref set and stays conservative-stale) diff --git a/src/kernel/__tests__/platform-collapse-parity.test.ts b/src/kernel/__tests__/platform-collapse-parity.test.ts index f5ee1ebd8..7abf6ec4b 100644 --- a/src/kernel/__tests__/platform-collapse-parity.test.ts +++ b/src/kernel/__tests__/platform-collapse-parity.test.ts @@ -1,8 +1,5 @@ import { test } from 'vitest'; import assert from 'node:assert/strict'; -import fs from 'node:fs'; -import os from 'node:os'; -import path from 'node:path'; import { deviceFieldsFromPublicPlatform, isIosFamily, @@ -24,8 +21,7 @@ import { VISIONOS_SIMULATOR, WEB_DESKTOP_DEVICE, } from '../../__tests__/test-utils/device-fixtures.ts'; -import { readReplayScriptMetadata, writeReplayScript } from '../../replay/script.ts'; -import type { SessionState } from '../../daemon/types.ts'; +import { readReplayScriptMetadata } from '../../replay/script.ts'; // Parity gate for the ios/macos -> apple Platform collapse (issue #979, approach b). // Internal `DeviceInfo.platform` is `apple`; the daemon still ACCEPTS the legacy @@ -113,24 +109,19 @@ test('predicates preserve the pre-collapse platform families', () => { assert.equal(isMobilePlatform(LINUX_DEVICE), false); }); -test('REPLAY: heal-write emits the leaf platform and round-trips through the reader', () => { - const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'platform-collapse-parity-')); +test('REPLAY: a context header built from publicPlatformString emits the leaf platform and round-trips through the reader', () => { + // Mirrors the live header assembly (daemon/session-script-writer.ts's + // formatScript: `context platform=${publicPlatformString(device)} ...`) — + // approach (b) writes the PUBLIC leaf platform (ios/macos), never the + // internal `apple`, so `.ad` scripts stay byte-compatible with checked-in + // fixtures and machine consumers. for (const device of [IOS_SIMULATOR, TVOS_SIMULATOR, MACOS_DEVICE, ANDROID_EMULATOR]) { - const session = { - name: 'parity', - device, - createdAt: 0, - actions: [], - } as unknown as SessionState; - const filePath = path.join(dir, `${device.id}.ad`); - writeReplayScript(filePath, [], session); - const written = fs.readFileSync(filePath, 'utf8'); const leaf = publicPlatformString(device); + const written = `context platform=${leaf} device="parity"\nopen "Demo"\n`; assert.match(written, new RegExp(`context platform=${leaf}\\b`), device.name); // The reader accepts the emitted leaf and echoes it back unchanged. assert.equal(readReplayScriptMetadata(written).platform, leaf, device.name); } // The reader also accepts the collapsed `apple` selector directly. assert.equal(readReplayScriptMetadata('context platform=apple\nhome\n').platform, 'apple'); - fs.rmSync(dir, { recursive: true, force: true }); }); diff --git a/src/replay/__tests__/divergence.test.ts b/src/replay/__tests__/divergence.test.ts index 91ecb7fed..204190a50 100644 --- a/src/replay/__tests__/divergence.test.ts +++ b/src/replay/__tests__/divergence.test.ts @@ -8,7 +8,6 @@ import { REPLAY_DIVERGENCE_DEFAULT_REF_LIMIT, REPLAY_DIVERGENCE_DIGEST_REF_LIMIT, REPLAY_DIVERGENCE_LEVEL_BYTE_LIMITS, - REPLAY_DIVERGENCE_RESUME_NOT_SUPPORTED, REPLAY_DIVERGENCE_SUGGESTION_LIMIT, truncateUtf8Field, type ReplayDivergence, @@ -24,7 +23,7 @@ function buildDivergence(overrides: Partial = {}): ReplayDiver screen: { state: 'available', refsGeneration: 4, refs: [{ ref: 'e1', role: 'button' }] }, suggestions: [], suggestionCount: 0, - resume: REPLAY_DIVERGENCE_RESUME_NOT_SUPPORTED, + resume: { allowed: true, from: 3, planDigest: 'deadbeef' }, ...overrides, }; } @@ -194,10 +193,16 @@ test('boundReplayDivergence sets artifactUnavailable when the artifact write its assert.ok(divergence.cause.message.startsWith(bounded.cause.message.slice(0, 50))); }); -test('resume is always allowed:false with a clear reason and carries no planDigest key at this migration step', () => { - const divergence = buildDivergence(); - assert.deepEqual(divergence.resume, { allowed: false, reason: 'resume not yet supported' }); - assert.ok(!('planDigest' in divergence.resume)); +test('resume carries the from index and planDigest through unmodified', () => { + const divergence = buildDivergence({ + resume: { allowed: false, from: 3, planDigest: 'deadbeef', reason: 'skips control flow' }, + }); + assert.deepEqual(divergence.resume, { + allowed: false, + from: 3, + planDigest: 'deadbeef', + reason: 'skips control flow', + }); }); // --- ADR 0012: redact BEFORE truncation ("All rendered strings ... pass @@ -249,7 +254,7 @@ test('formatReplayDivergenceReport lists a bounded ref/role/label subset for an }, suggestions: [], suggestionCount: 0, - resume: { allowed: false, reason: 'resume not yet supported' }, + resume: { allowed: true, from: 5, planDigest: 'deadbeef' }, }, }); assert.ok(report); @@ -278,7 +283,7 @@ test('formatReplayDivergenceReport carries the unavailable-screen hint', async ( }, suggestions: [], suggestionCount: 0, - resume: { allowed: false, reason: 'resume not yet supported' }, + resume: { allowed: true, from: 5, planDigest: 'deadbeef' }, }, }); assert.ok(report); diff --git a/src/replay/__tests__/plan-digest.test.ts b/src/replay/__tests__/plan-digest.test.ts new file mode 100644 index 000000000..2a9fb611d --- /dev/null +++ b/src/replay/__tests__/plan-digest.test.ts @@ -0,0 +1,164 @@ +import assert from 'node:assert/strict'; +import { test } from 'vitest'; +import { computeReplayPlanDigest } from '../plan-digest.ts'; +import type { SessionAction } from '../../daemon/types.ts'; + +function action(overrides: Partial = {}): SessionAction { + return { + ts: 0, + command: 'click', + positionals: ['label="Save"'], + flags: {}, + ...overrides, + }; +} + +function digestFor( + actions: SessionAction[], + overrides: Partial[0]> = {}, +) { + return computeReplayPlanDigest({ + actions, + actionLines: actions.map((_, i) => i + 1), + actionSourcePaths: undefined, + metadata: {}, + ...overrides, + }); +} + +test('computeReplayPlanDigest is a 64-char lowercase hex SHA-256', () => { + const digest = digestFor([action()]); + assert.match(digest, /^[0-9a-f]{64}$/); +}); + +test('computeReplayPlanDigest is stable across repeated computation of the same plan', () => { + const actions = [action(), action({ command: 'open', positionals: ['Demo'] })]; + assert.equal(digestFor(actions), digestFor(actions)); +}); + +test('computeReplayPlanDigest is insensitive to incidental flags-object key order', () => { + const a = action({ flags: { settle: true, timeoutMs: 500 } as never }); + const b = action({ flags: { timeoutMs: 500, settle: true } as never }); + assert.equal(digestFor([a]), digestFor([b])); +}); + +test('computeReplayPlanDigest changes when a positional changes', () => { + const original = digestFor([action({ positionals: ['label="Save"'] })]); + const edited = digestFor([action({ positionals: ['label="Submit"'] })]); + assert.notEqual(original, edited); +}); + +test('computeReplayPlanDigest changes when the command changes', () => { + const original = digestFor([action({ command: 'click' })]); + const edited = digestFor([action({ command: 'press' })]); + assert.notEqual(original, edited); +}); + +test('computeReplayPlanDigest changes when source provenance (path/line) changes', () => { + const actions = [action()]; + const original = digestFor(actions, { actionLines: [1] }); + const movedLine = digestFor(actions, { actionLines: [5] }); + assert.notEqual(original, movedLine); + + const noSourcePath = digestFor(actions, { actionSourcePaths: undefined }); + const withSourcePath = digestFor(actions, { actionSourcePaths: ['/tmp/include.ad'] }); + assert.notEqual(noSourcePath, withSourcePath); +}); + +test('computeReplayPlanDigest changes when platform-conditioned metadata changes', () => { + const actions = [action()]; + const ios = digestFor(actions, { metadata: { platform: 'ios' } }); + const android = digestFor(actions, { metadata: { platform: 'android' } }); + assert.notEqual(ios, android); +}); + +test('computeReplayPlanDigest folds runtime control-flow shape (retry) into the digest', () => { + const retryOne: SessionAction = action({ + command: 'back', + positionals: [], + replayControl: { kind: 'retry', maxRetries: 1, actions: [action({ command: 'back' })] }, + }); + const retryTwo: SessionAction = { + ...retryOne, + replayControl: { kind: 'retry', maxRetries: 2, actions: [action({ command: 'back' })] }, + }; + assert.notEqual(digestFor([retryOne]), digestFor([retryTwo])); +}); + +test('computeReplayPlanDigest folds runtime control-flow shape (maestroRunFlowWhen) into the digest', () => { + const base: SessionAction = action({ + command: 'back', + positionals: [], + replayControl: { + kind: 'maestroRunFlowWhen', + mode: 'visible', + selector: 'label="Continue"', + actions: [action({ command: 'back' })], + }, + }); + const changedSelector: SessionAction = { + ...base, + replayControl: { + kind: 'maestroRunFlowWhen', + mode: 'visible', + selector: 'label="Skip"', + actions: [action({ command: 'back' })], + }, + }; + assert.notEqual(digestFor([base]), digestFor([changedSelector])); +}); + +test('computeReplayPlanDigest changes when an execution runtime hint changes', () => { + const runtime = { + platform: 'ios' as const, + metroHost: '127.0.0.1', + metroPort: 8081, + bundleUrl: 'http://localhost:8081/index.bundle', + launchUrl: 'agentdevice://home', + }; + const original = digestFor([action({ runtime })]); + + for (const [key, value] of Object.entries({ + platform: 'android', + metroHost: '10.0.2.2', + metroPort: 8082, + bundleUrl: 'http://localhost:8082/index.bundle', + launchUrl: 'agentdevice://settings', + })) { + assert.notEqual(original, digestFor([action({ runtime: { ...runtime, [key]: value } })])); + } +}); + +test('computeReplayPlanDigest changes when target evidence consumed before action changes', () => { + const targetEvidence = { + id: 'save', + role: 'button', + label: 'Save', + ancestry: [], + sibling: 0, + viewportOrder: 0, + verification: 'verified' as const, + }; + const original = digestFor([action({ targetEvidence })]); + + for (const targetEvidenceChange of [ + { label: 'Submit' }, + { sibling: 1 }, + { viewportOrder: 1 }, + { verification: 'unverifiable' as const }, + ]) { + assert.notEqual( + original, + digestFor([action({ targetEvidence: { ...targetEvidence, ...targetEvidenceChange } })]), + ); + } +}); + +test('computeReplayPlanDigest never changes based on unsubstituted ${VAR} text (variable VALUES never affect the digest)', () => { + // The digest is computed over the still-unsubstituted action text; --env + // values are resolved later, at invocation time, so two runs with + // different --env inputs against the identical script must agree. + const withVar = digestFor([action({ positionals: ['label="${NAME}"'] })]); + const sameWithVar = digestFor([action({ positionals: ['label="${NAME}"'] })]); + assert.equal(withVar, sameWithVar); +}); diff --git a/src/replay/__tests__/script.test.ts b/src/replay/__tests__/script.test.ts index df8fcb388..6f4ed1830 100644 --- a/src/replay/__tests__/script.test.ts +++ b/src/replay/__tests__/script.test.ts @@ -1,36 +1,32 @@ import { test } from 'vitest'; import assert from 'node:assert/strict'; -import fs from 'node:fs'; -import os from 'node:os'; -import path from 'node:path'; import { AppError } from '../../kernel/errors.ts'; import { parseReplayScriptDetailed, readReplayScriptMetadata, REPLAY_METADATA_PLATFORMS, - writeReplayScript, } from '../script.ts'; +import { formatPortableActionLine, formatTargetAnnotationLines } from '../script-formatting.ts'; import type { TargetAnnotationV1 } from '../target-identity.ts'; -import type { SessionAction, SessionState } from '../../daemon/types.ts'; - -function makeSession(): SessionState { - return { - name: 'default', - device: { - platform: 'android', - id: 'emulator-5554', - name: 'Pixel', - kind: 'emulator', - booted: true, - }, - createdAt: Date.now(), - actions: [], - }; +import type { SessionAction } from '../../daemon/types.ts'; + +// `writeReplayScript` (the `--update` heal-and-rewrite serializer) was +// deleted in ADR 0012 migration step 6 (`--update` retirement left it with +// zero production consumers). These tests exercise the underlying +// line-formatting primitives it used to call — still live, still shared with +// the session recorder's own writer (`daemon/session-script-writer.ts`) — +// directly, plus `parseReplayScriptDetailed` for round-trip proof. + +function formatReplayScriptForTest(actions: SessionAction[]): string { + const lines: string[] = []; + for (const action of actions) { + lines.push(...formatTargetAnnotationLines(action)); + lines.push(formatPortableActionLine(action, { runtimeIncludeAllPositionals: true })); + } + return `${lines.join('\n')}\n`; } -test('writeReplayScript preserves inline open runtime hints', () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-script-open-')); - const replayPath = path.join(root, 'flow.ad'); +test('formatPortableActionLine preserves inline open runtime hints', () => { const actions: SessionAction[] = [ { ts: Date.now(), @@ -46,8 +42,7 @@ test('writeReplayScript preserves inline open runtime hints', () => { }, ]; - writeReplayScript(replayPath, actions, makeSession()); - const script = fs.readFileSync(replayPath, 'utf8'); + const script = formatReplayScriptForTest(actions); assert.match( script, @@ -68,8 +63,6 @@ test('record replay script parses fps, max-size, quality, and hide-touches flags }); test('screenshot replay script round-trips screenshot flags', () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-script-screenshot-')); - const replayPath = path.join(root, 'flow.ad'); const actions: SessionAction[] = [ { ts: Date.now(), @@ -84,8 +77,7 @@ test('screenshot replay script round-trips screenshot flags', () => { }, ]; - writeReplayScript(replayPath, actions, makeSession()); - const script = fs.readFileSync(replayPath, 'utf8'); + const script = formatReplayScriptForTest(actions); assert.match( script, /screenshot "\.\/page\.png" --pixel-density 2 --fullscreen --max-size 1024 --no-stabilize/, @@ -123,8 +115,6 @@ test('snapshot replay script parses full refresh flags', () => { }); test('snapshot replay script writes interactive refresh flags', () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-script-snapshot-')); - const replayPath = path.join(root, 'flow.ad'); const actions: SessionAction[] = [ { ts: Date.now(), @@ -138,8 +128,7 @@ test('snapshot replay script writes interactive refresh flags', () => { }, ]; - writeReplayScript(replayPath, actions, makeSession()); - const script = fs.readFileSync(replayPath, 'utf8'); + const script = formatReplayScriptForTest(actions); assert.match(script, /snapshot -i -d 2 -s @e1/); }); @@ -169,8 +158,6 @@ test('gesture replay script parses pan, fling, swipe, pinch, and rotate gesture }); test('type and fill replay scripts round-trip typing delay flags', () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-script-typing-')); - const replayPath = path.join(root, 'flow.ad'); const actions: SessionAction[] = [ { ts: Date.now(), @@ -186,8 +173,7 @@ test('type and fill replay scripts round-trip typing delay flags', () => { }, ]; - writeReplayScript(replayPath, actions, makeSession()); - const script = fs.readFileSync(replayPath, 'utf8'); + const script = formatReplayScriptForTest(actions); assert.match(script, /type "hello world" --delay-ms 75/); assert.match(script, /fill @e2 "search" --delay-ms 40/); @@ -202,24 +188,21 @@ test('type replay script preserves literal delay flag tokens', () => { assert.equal(parsed[0]?.flags.delayMs, undefined); }); -test('writeReplayScript escapes device labels with quotes and backslashes', () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-script-device-label-')); - const replayPath = path.join(root, 'flow.ad'); - const session = makeSession(); - session.device.name = 'Pixel "QA" \\ Lab'; - - writeReplayScript(replayPath, [], session); - const script = fs.readFileSync(replayPath, 'utf8'); - - assert.match( - script, - /context platform=android device="Pixel \\"QA\\" \\\\ Lab" kind=emulator theme=unknown/, +test('formatScriptStringLiteral escapes device labels with quotes and backslashes for a context header', async () => { + const { formatScriptStringLiteral } = await import('../script-utils.ts'); + // Same assembly the live session-script-writer uses + // (daemon/session-script-writer.ts's formatScript): `context platform=... + // device= kind=... theme=...`. + const header = `context platform=android device=${formatScriptStringLiteral('Pixel "QA" \\ Lab')} kind=emulator theme=unknown`; + assert.equal( + header, + 'context platform=android device="Pixel \\"QA\\" \\\\ Lab" kind=emulator theme=unknown', ); + // And it round-trips through the reader as ordinary context metadata. + assert.equal(readReplayScriptMetadata(`${header}\nopen "Demo"\n`).platform, 'android'); }); -test('writeReplayScript preserves significant whitespace and empty string arguments', () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-script-whitespace-')); - const replayPath = path.join(root, 'flow.ad'); +test('a rewritten script preserves significant whitespace and empty string arguments', () => { const actions: SessionAction[] = [ { ts: Date.now(), @@ -258,8 +241,7 @@ test('writeReplayScript preserves significant whitespace and empty string argume }, ]; - writeReplayScript(replayPath, actions, makeSession()); - const script = fs.readFileSync(replayPath, 'utf8'); + const script = formatReplayScriptForTest(actions); assert.match(script, /type " leading\\ttrailing "/); assert.match(script, /fill @e2 ""/); @@ -451,9 +433,7 @@ test('old readers ignoring the comment execute the action unchanged: an action w assert.equal(actions[0]?.targetEvidence, undefined); }); -test('writeReplayScript (read-then-rewrite / heal path) preserves a v1 annotation in canonical form', () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-script-target-v1-')); - const replayPath = path.join(root, 'flow.ad'); +test('a rewritten script preserves a v1 annotation in canonical form', () => { const actions: SessionAction[] = [ { ts: Date.now(), @@ -464,8 +444,7 @@ test('writeReplayScript (read-then-rewrite / heal path) preserves a v1 annotatio }, ]; - writeReplayScript(replayPath, actions, makeSession()); - const script = fs.readFileSync(replayPath, 'utf8'); + const script = formatReplayScriptForTest(actions); const lines = script.trim().split('\n'); assert.equal(lines.at(-2), SAVE_EVIDENCE_LINE); assert.equal(lines.at(-1), 'click @e12'); @@ -476,13 +455,10 @@ test('writeReplayScript (read-then-rewrite / heal path) preserves a v1 annotatio }); test('session-recorded actions without target evidence never gain a fabricated annotation on rewrite', () => { - const root = fs.mkdtempSync(path.join(os.tmpdir(), 'agent-device-replay-script-target-v1-none-')); - const replayPath = path.join(root, 'flow.ad'); const actions: SessionAction[] = [ { ts: Date.now(), command: 'click', positionals: ['@e12'], flags: {} }, ]; - writeReplayScript(replayPath, actions, makeSession()); - const script = fs.readFileSync(replayPath, 'utf8'); + const script = formatReplayScriptForTest(actions); assert.equal(/agent-device:target-v1/.test(script), false); }); diff --git a/src/replay/divergence.ts b/src/replay/divergence.ts index 7f502b744..27991283b 100644 --- a/src/replay/divergence.ts +++ b/src/replay/divergence.ts @@ -64,11 +64,17 @@ export type ReplayDivergenceSuggestion = { label?: string; }; -/** Always `allowed: false` until migration step 5; `from`/`planDigest` keys absent until then. */ -export type ReplayDivergenceResume = { - allowed: false; - reason: string; -}; +/** + * ADR 0012 decision 4 / migration step 5. `from` is the failed step's 1-based + * plan index and `planDigest` is the SHA-256 digest of the canonical fully + * expanded plan (`computeReplayPlanDigest`) that produced this report — both + * are always present. `allowed` is the preflight verdict for resuming AT + * `from` (`evaluateReplayResumePreflight`); `reason` is present only when + * `allowed` is `false`. + */ +export type ReplayDivergenceResume = + | { allowed: true; from: number; planDigest: string } + | { allowed: false; from: number; planDigest: string; reason: string }; export type ReplayDivergenceOverflow = { omittedBytes: number; @@ -90,11 +96,6 @@ export type ReplayDivergence = { artifactUnavailable?: true; }; -export const REPLAY_DIVERGENCE_RESUME_NOT_SUPPORTED: ReplayDivergenceResume = { - allowed: false, - reason: 'resume not yet supported', -}; - type BoundedResponseLevel = 'digest' | 'default' | 'full'; export const REPLAY_DIVERGENCE_LEVEL_BYTE_LIMITS: Record = { diff --git a/src/replay/plan-digest.ts b/src/replay/plan-digest.ts new file mode 100644 index 000000000..6d08ccb9a --- /dev/null +++ b/src/replay/plan-digest.ts @@ -0,0 +1,103 @@ +import { createHash } from 'node:crypto'; +import type { SessionAction } from '../daemon/types.ts'; + +/** + * ADR 0012 decision 4 / migration step 5: `planDigest` is SHA-256 over the + * canonical fully expanded plan — the SAME `actions` array the replay + * runtime iterates, already flattened at parse time (static includes, + * platform conditions, and fixed-count repeats expand before this point; + * see `parseReplayInput`/`parseMaestroReplayFlow`). Runtime-only control flow + * (`retry`, `maestroRunFlowWhen`) stays a single plan entry — its shape + * (kind/mode/selector/maxRetries and nested action shapes) is folded into the + * digest so an edit inside a control-flow block still changes the digest, + * even though its nested actions are never individually addressable by + * `--from`. + * + * Deliberately excluded: values resolved at invocation time. Variable + * substitution happens in `resolveReplayAction`, after this digest is + * computed over the still-unsubstituted `${VAR}` text, so `--env`/shell + * values never change the digest. Every parsed value consumed before or + * during action execution, including runtime hints and target evidence, is + * part of the canonical plan. + */ +export type ReplayPlanDigestMetadata = { + platform?: string; + target?: string; +}; + +export function computeReplayPlanDigest(params: { + actions: SessionAction[]; + actionLines: number[]; + actionSourcePaths?: (string | undefined)[]; + metadata: ReplayPlanDigestMetadata; +}): string { + const canonical = buildCanonicalPlan(params); + const json = stableStringify(canonical); + return createHash('sha256').update(json, 'utf8').digest('hex'); +} + +function buildCanonicalPlan(params: { + actions: SessionAction[]; + actionLines: number[]; + actionSourcePaths?: (string | undefined)[]; + metadata: ReplayPlanDigestMetadata; +}): unknown { + const { actions, actionLines, actionSourcePaths, metadata } = params; + return { + platform: metadata.platform ?? null, + target: metadata.target ?? null, + steps: actions.map((action, index) => + canonicalizeAction(action, actionLines[index] ?? null, actionSourcePaths?.[index] ?? null), + ), + }; +} + +function canonicalizeAction( + action: SessionAction, + line: number | null, + sourcePath: string | null, +): unknown { + return { + command: action.command, + positionals: action.positionals ?? [], + flags: action.flags ?? {}, + runtime: action.runtime ?? null, + control: canonicalizeControl(action.replayControl), + targetEvidence: action.targetEvidence ?? null, + source: { path: sourcePath, line }, + }; +} + +function canonicalizeControl(control: SessionAction['replayControl']): unknown { + if (!control) return null; + const nested = control.actions.map((nestedAction, index) => { + const source = control.actionSources?.[index]; + return canonicalizeAction(nestedAction, source?.line ?? null, source?.path ?? null); + }); + if (control.kind === 'retry') { + return { kind: control.kind, maxRetries: control.maxRetries, actions: nested }; + } + return { kind: control.kind, mode: control.mode, selector: control.selector, actions: nested }; +} + +/** + * Deterministic JSON serialization: object keys are sorted so the digest + * never depends on incidental property insertion order (e.g. how an action's + * `flags` bag was built up across parse/normalization steps). + */ +function stableStringify(value: unknown): string { + return JSON.stringify(sortKeysDeep(value)); +} + +function sortKeysDeep(value: unknown): unknown { + if (Array.isArray(value)) return value.map(sortKeysDeep); + if (value && typeof value === 'object') { + const record = value as Record; + const sorted: Record = {}; + for (const key of Object.keys(record).sort()) { + sorted[key] = sortKeysDeep(record[key]); + } + return sorted; + } + return value; +} diff --git a/src/replay/script.ts b/src/replay/script.ts index d914ca7dd..1c0ee7db6 100644 --- a/src/replay/script.ts +++ b/src/replay/script.ts @@ -1,14 +1,11 @@ -import fs from 'node:fs'; import { AppError } from '../kernel/errors.ts'; import { recordingQualityInputToExportQuality } from '../core/recording-export-quality.ts'; import { readScreenshotScriptFlag } from '../contracts/screenshot.ts'; import type { DeviceTarget, PlatformSelector } from '../kernel/device.ts'; -import { PLATFORM_SELECTORS, publicPlatformString } from '../kernel/device.ts'; +import { PLATFORM_SELECTORS } from '../kernel/device.ts'; import { parseReplayOpenFlags } from './open-script.ts'; -import { formatPortableActionLine, formatTargetAnnotationLines } from './script-formatting.ts'; -import type { SessionAction, SessionState } from '../daemon/types.ts'; +import type { SessionAction } from '../daemon/types.ts'; import { - formatScriptStringLiteral, isClickLikeCommand, parseReplaySeriesFlags, parseReplayRuntimeFlags, @@ -507,36 +504,3 @@ function readBareReplayToken(line: string, cursor: number): { value: string; nex } return { value: line.slice(cursor, end), nextCursor: end }; } - -export function writeReplayScript( - filePath: string, - actions: SessionAction[], - session?: SessionState, -) { - const lines: string[] = []; - // Session can be missing if the replay session is closed/deleted between execution and update write. - // In that case we still persist healed actions and omit only the context header. - if (session) { - const kind = session.device.kind ? ` kind=${session.device.kind}` : ''; - const target = session.device.target ? ` target=${session.device.target}` : ''; - // approach (b): heal-write the PUBLIC leaf platform (ios/macos), never the - // internal `apple` — keeps healed `.ad` scripts byte-compatible with checked-in - // fixtures and machine consumers. - lines.push( - `context platform=${publicPlatformString(session.device)}${target} device=${formatScriptStringLiteral(session.device.name)}${kind} theme=unknown`, - ); - } - for (const action of actions) { - // ADR 0012 decision 3: rewrites preserve v1 annotations in canonical form. - lines.push(...formatTargetAnnotationLines(action)); - lines.push(formatReplayActionLine(action)); - } - const serialized = `${lines.join('\n')}\n`; - const tmpPath = `${filePath}.tmp-${process.pid}-${Date.now()}`; - fs.writeFileSync(tmpPath, serialized); - fs.renameSync(tmpPath, filePath); -} - -function formatReplayActionLine(action: SessionAction): string { - return formatPortableActionLine(action, { runtimeIncludeAllPositionals: true }); -} diff --git a/test/integration/provider-scenarios/android-lifecycle.test.ts b/test/integration/provider-scenarios/android-lifecycle.test.ts index 9f21843bc..4625f952d 100644 --- a/test/integration/provider-scenarios/android-lifecycle.test.ts +++ b/test/integration/provider-scenarios/android-lifecycle.test.ts @@ -1144,6 +1144,36 @@ async function runAndroidCaptureInteractionAndReplayWorkflow( }); assert.equal(replayEnv.replayed, 3); + // ADR 0012 step 5: replay resume. A full run diverges at step 2 (a + // selector matching nothing); the report's resume object carries the plan + // digest. Resuming at the next index after the failed step (the documented + // "completed the failed action manually" loop) skips steps 1-2 without + // executing them and replays only the tail. + const resumePath = path.join(tempRoot, 'settings-resume.ad'); + fs.writeFileSync( + resumePath, + ['snapshot -i', 'press label="NoSuchControl"', 'get text @e3 Search', ''].join('\n'), + ); + const divergenceError = await client.replay.run({ path: resumePath, ...selection }).then( + () => null, + (error: unknown) => error as { code?: string; details?: Record }, + ); + assert.ok(divergenceError, 'expected the replay to diverge on the missing selector'); + assert.equal(divergenceError.code, 'REPLAY_DIVERGENCE'); + const divergenceReport = divergenceError.details?.divergence as { + resume: { allowed: boolean; from: number; planDigest: string }; + }; + assert.equal(divergenceReport.resume.allowed, true); + assert.equal(divergenceReport.resume.from, 2); + assert.match(divergenceReport.resume.planDigest, /^[0-9a-f]{64}$/); + const resumedReplay = await client.replay.run({ + path: resumePath, + resumeFrom: divergenceReport.resume.from + 1, + resumePlanDigest: divergenceReport.resume.planDigest, + ...selection, + }); + assert.equal(resumedReplay.replayed, 1); + const screenshot = await client.capture.screenshot({ path: screenshotPath, ...selection, diff --git a/test/skillgym/suites/agent-device-smoke-suite.ts b/test/skillgym/suites/agent-device-smoke-suite.ts index 4ac0ff653..e7d94d066 100644 --- a/test/skillgym/suites/agent-device-smoke-suite.ts +++ b/test/skillgym/suites/agent-device-smoke-suite.ts @@ -2493,11 +2493,22 @@ Treat the recovery message as a warning, not a fatal error. Use the exposed Sear contract: [ 'Replay path: ./replays/catalog-checkout.ad', 'Selectors drifted after a UI label change', - 'Goal: maintain the replay script in place', + '--update is accepted but retired: it never rewrites the replay file and reports healed: 0', + 'The divergence reports resume.from=4 and resume.planDigest=0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef', + 'Workflow A: after accepting a suggestion and editing the script, the old digest is stale; run a fresh full replay with no resume flags', + 'Workflow B: when the script and includes stay unchanged, repair app state so retrying the failed action is safe, then resume with the reported values', + 'Output exactly the two alternative replay commands in A-then-B order; manual edits and app-state repairs happen out of band', + ], + task: 'Plan both valid replay-maintenance commands after selector drift.', + outputs: [ + /(?:^|\n)agent-device\s+replay\s+\.\/replays\/catalog-checkout\.ad\s*(?:\n|$)/i, + /(?:^|\n)agent-device\s+replay\s+\.\/replays\/catalog-checkout\.ad(?=[^\n]*--from\s+4\b)(?=[^\n]*--plan-digest\s+0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\b)[^\n]*(?:\n|$)/i, + ], + forbiddenOutputs: [ + /(?:^|\n)agent-device\s+replay\s+\.\/replays\/catalog-checkout\.ad[^\n]*(?:-u\b|--update\b)/i, + /(?:^|\n)agent-device\s+replay\s+\.\/replays\/catalog-checkout\.ad(?=[^\n]*--from\b)(?![^\n]*--plan-digest\b)[^\n]*/i, + /(?:^|\n)agent-device\s+replay\s+\.\/replays\/catalog-checkout\.ad(?=[^\n]*--plan-digest\b)(?![^\n]*--from\b)[^\n]*/i, ], - task: 'Plan the command to maintain the existing replay script after selector drift.', - outputs: [plannedCommand('replay'), /-u|--update/i, /\.\/replays\/catalog-checkout\.ad/i], - forbiddenOutputs: [/sed\s+-i/i, /open .*\.ad/i], }), makeCase({ id: 'replay-maestro-compatibility-flow', diff --git a/website/docs/docs/commands.md b/website/docs/docs/commands.md index 536c2c55e..e2d0df9f8 100644 --- a/website/docs/docs/commands.md +++ b/website/docs/docs/commands.md @@ -421,7 +421,7 @@ agent-device open Settings --platform ios --session e2e --save-script [path] agent-device replay ./session.ad # Run deterministic replay from .ad script agent-device test ./suite # Run every .ad file in a folder or glob serially agent-device test ./suite --timeout 60000 --retries 1 -agent-device replay -u ./session.ad # Update selector drift and rewrite .ad script in place +agent-device replay ./session.ad --from 4 --plan-digest # Execute step 4; if already completed, use the next safe index with this digest ``` - `replay` runs deterministic `.ad` scripts. @@ -430,7 +430,8 @@ agent-device replay -u ./session.ad # Update selector drift and rewrite .ad sc - `test --timeout ` and `test --retries ` apply per script attempt; `context timeout=...` and `context retries=...` can be declared inside the `.ad` header. Retries are capped at `3`, duplicate metadata keys are rejected, and timeouts are cooperative. - `test --artifacts-dir ` overrides the default suite artifact root at `.agent-device/test-artifacts`. - `test` prints a short `Running replay suite...` line before dispatch, then streams one-line `pass`, `fail`, or `skip` progress on stderr as each suite entry finishes or retries. Each line includes current/total suite position and elapsed seconds such as `pass 3/6 ... duration=12.34s`. The final summary still prints failures and flaky passed-on-retry tests by default; add `--verbose` to print every final result. -- `replay -u` updates stale recorded actions and rewrites the same script. +- A failing step returns a `REPLAY_DIVERGENCE` report (screen digest, ranked selector suggestions, and a `resume` field); `replay --from --plan-digest ` resumes at and executes plan step `n` without re-running `1..n-1`. If the failed action was completed manually, resume from the next safe plan index using the matching digest. `replay`-only; `test` rejects `--from`. +- `replay -u`/`--update` no longer rewrites the script (retired — see [Replay & E2E](/docs/replay-e2e)); it is a no-op kept for compatibility, since every divergence already carries the same ranked suggestions. - `--save-script` records a replay script on `close`; optional path is a file path and parent directories are created. See [Replay & E2E](/docs/replay-e2e) for recording, Maestro compatibility, and CI workflow details. diff --git a/website/docs/docs/replay-e2e.md b/website/docs/docs/replay-e2e.md index 70ff506e4..66f88acd7 100644 --- a/website/docs/docs/replay-e2e.md +++ b/website/docs/docs/replay-e2e.md @@ -283,64 +283,74 @@ Quote `${VAR}` inside selector expressions so the whole expression is treated as ### Notes -- `replay -u` does not yet preserve `env` directives or `${VAR}` tokens. Workaround: temporarily inline the literal values, run `-u`, re-parametrise. - Shell env (`AD_VAR_*`) is collected on the CLI/client side at request time, so the same values are seen whether the daemon runs locally or remotely. - No nested fallback. `${A:-${B}}` is not supported. - Unresolved `${VAR}` fails with a `file:line` reference. Typos are loud. -## Update stale selectors in replay scripts - -```bash -agent-device replay -u ~/.agent-device/sessions/e2e-2026-02-09T12-00-00-000Z.ad --session e2e-run +## Replay divergence and resume + +A failing `replay`/`test` step returns a structured `REPLAY_DIVERGENCE` error instead of a bare failure. The report carries, bounded and redacted: + +- **`step`** — the 1-based plan index and its source file/line (through Maestro `runFlow` includes). +- **`screen`** — a fresh post-failure snapshot digest with actionable refs, or `unavailable` with a reason/hint when capture failed or was sparse (never a stale tree). +- **`suggestions`** — up to 5 ranked, re-resolved candidates for the failing selector (id match ranks above role+label, which ranks above label-only), each with a `basis` you can inspect before acting. +- **`resume`** — whether and how to continue without re-running the script from the top. + +```jsonc +{ + "code": "REPLAY_DIVERGENCE", + "details": { + "divergence": { + "step": { "index": 4, "source": { "path": "flow.ad", "line": 6 } }, + "screen": { "state": "available", "refsGeneration": 3, "refs": [ /* ... */ ] }, + "suggestions": [{ "selector": "id=\"auth_continue\"", "basis": "id" }], + "resume": { "allowed": true, "from": 4, "planDigest": "…64 hex chars…" } + } + } +} ``` -When a replay step fails, update can: +Text output prints a compact summary of the same fields; `--json`/MCP carry the full object. -- Take a fresh snapshot. -- Resolve a stable replacement target. -- Retry the step. -- Rewrite the failing line in the same `.ad` file. +### Resuming a failed replay -Current update targets: +`replay --from --plan-digest ` resumes **at** plan step `n`, not after it, skipping `1..n-1` without executing them. Both flags come from a divergence report's `resume` field — `from` is the failed step, `planDigest` is the digest of the exact unchanged plan that produced it. -- `click` -- `fill` -- `get` -- `is` -- `wait` +Choose one recovery workflow: -## `replay -u` before/after examples +1. **Change the replay script.** Review the suggestion, edit the selector or include, then run a fresh full `replay ./flow.ad`. The old digest is intentionally invalid after any plan edit; do not combine it with the edited script. A later divergence supplies a new digest. +2. **Keep the replay plan unchanged.** Repair app/device state so the reported failed step can succeed when retried, then resume with the report's unchanged `from` and `planDigest`. If you manually complete the failed action itself, the reported `from` will execute it again; only do that when repeating the action is safe. -Example 1: stale selector rewritten in place +The unchanged-plan resume loop is: -```sh -# Before -click "id=\"old_continue\" || label=\"Continue\"" +1. Run `replay ./flow.ad`. On failure, read `resume` from the divergence. +2. Leave the script, includes, platform, and target unchanged. Repair app state yourself so the failed step can be retried safely. The daemon never infers or reconstructs app state — it only skips execution of the earlier steps. +3. `replay ./flow.ad --from --plan-digest `. -# After `replay -u` -click "id=\"auth_continue\" || label=\"Continue\"" +```bash +agent-device replay ./flow.ad +# ... REPLAY_DIVERGENCE, resume: { allowed: true, from: 4, planDigest: "ab12...“ } +# (repair app state on the device) +agent-device replay ./flow.ad --from 4 --plan-digest ab12... ``` -Example 2: stale ref-based action upgraded to selector form +`resume.allowed` is `false`, with a `reason`, when resuming cannot be proven safe: -```sh -# Before -snapshot -i -s "Continue" -click @e13 "Continue" +- a skipped step can produce `outputEnv` values (a Maestro `runScript` step) a later step might consume; +- the skipped range or the resume target itself is runtime control flow (a Maestro `retry:` or `runFlow.when:` block) — these execute dynamically and are never individually addressable by `--from`. -# After `replay -u` -snapshot -i -s "Continue" -click "id=\"auth_continue\" || label=\"Continue\"" -``` +Passing `--plan-digest` that no longer matches the current script — because you edited it, an include changed, or platform-conditioned expansion differs — fails `INVALID_ARGS` before any action; run a fresh full replay to get a new digest. `--from` is `replay`-only; `test` rejects it (a suite run must stay full and deterministic). + +## `--update`/`-u` (retired) -Use `replay -u` locally during maintenance, review the rewritten `.ad` lines, then commit the updated script. +`--update`/`-u` no longer rewrites `.ad` files. Historically it retried a failing step against the recorded selector's candidate material and rewrote the line in place; the audit behind [ADR 0012](https://github.com/callstack/agent-device/blob/main/docs/adr/0012-interactive-replay.md) found that mechanism rarely able to act (it can only recover drift the original selector still matches, never a rename) and a silent rewrite is a target-binding risk on its own. The flag is kept, accepted, and is a complete no-op: every replay divergence already carries the same ranked `suggestions` the old heal path used to apply blind, whether or not `--update` is passed. Review a suggestion, then edit the `.ad` file yourself if it's right. ## Troubleshooting - Replay fails after UI/layout changes: - - Run `replay -u` locally and review the rewritten lines. -- Updating cannot resolve a unique target: - - Re-record that flow (`--save-script`) from a fresh exploratory pass. + - Read the divergence report's `suggestions` and repair the selector by hand; there is no automated rewrite. Because the edit changes the plan digest, run a fresh full replay instead of using the old resume flags. +- Repeated re-runs are slow or the app is stateful, but the script is still correct: + - Leave the replay plan unchanged, repair app state so the reported failed step can be retried, then use its `--from`/`--plan-digest`. Resume starts at `--from`; it does not skip that step. - Replay file parse error: - Validate quoting in `.ad` lines (unclosed quotes are rejected). - Maestro compatibility flow fails on unsupported syntax: