Problem
PR #1174 added deterministic per-surface output budgets and actionability floors, including the
combined MCP text + structured payload. The remaining gap is a representative workflow-level
oracle: a smaller response can still cost more if it causes an extra observation, retry, or
re-orientation after an error.
Scope
Add a focused, recorded routine-workflow benchmark that pairs output cost with successful next-step
behavior:
snapshot -i and unchanged-snapshot suppression;
- one mutation with
--settle, including the unchanged-interactive tail;
get text or get attrs;
- one actionable timeout/platform/selector failure that retains enough identity and session state
for recovery;
- CLI and MCP projections where they differ.
Track stable fixture-level UTF-8 bytes plus command count, fallback observation count, retry count,
and whether recovery preserves the current session. Keep live device measurements informational.
Add or extend small-model SkillGym/help-conformance cases so the outcome oracle verifies the next
command without a redundant observation. Do not make model-in-the-loop execution a required PR gate
unless the existing infrastructure is already deterministic and credential-free; the recorded
contract remains the fast CI gate.
Constraints
- Do not optimize bytes by removing actionable refs, generation pins, stable error identity, retry
guidance, or recovery handles.
- Do not gate on live simulator output sizes.
- Keep the work scoped to measurement and fixtures; response-default changes belong in a later,
evidence-driven issue.
Acceptance criteria
- A deterministic routine workflow reports output bytes and follow-up/retry counts.
- Regression tests fail when the workflow requires a redundant observation or loses recovery
fields.
- At least one small-model task covers the same next-step decision.
pnpm test:output-economy, relevant benchmark tests, typecheck, lint, and Fallow pass.
Origin: post-#1174 architecture follow-up.
Problem
PR #1174 added deterministic per-surface output budgets and actionability floors, including the
combined MCP text + structured payload. The remaining gap is a representative workflow-level
oracle: a smaller response can still cost more if it causes an extra observation, retry, or
re-orientation after an error.
Scope
Add a focused, recorded routine-workflow benchmark that pairs output cost with successful next-step
behavior:
snapshot -iand unchanged-snapshot suppression;--settle, including the unchanged-interactive tail;get textorget attrs;for recovery;
Track stable fixture-level UTF-8 bytes plus command count, fallback observation count, retry count,
and whether recovery preserves the current session. Keep live device measurements informational.
Add or extend small-model SkillGym/help-conformance cases so the outcome oracle verifies the next
command without a redundant observation. Do not make model-in-the-loop execution a required PR gate
unless the existing infrastructure is already deterministic and credential-free; the recorded
contract remains the fast CI gate.
Constraints
guidance, or recovery handles.
evidence-driven issue.
Acceptance criteria
fields.
pnpm test:output-economy, relevant benchmark tests, typecheck, lint, and Fallow pass.Origin: post-#1174 architecture follow-up.