Skip to content

Pair output budgets with workflow success and recovery benchmarks #1180

Description

@devin-ai-integration

Problem

PR #1174 added deterministic per-surface output budgets and actionability floors, including the
combined MCP text + structured payload. The remaining gap is a representative workflow-level
oracle: a smaller response can still cost more if it causes an extra observation, retry, or
re-orientation after an error.

Scope

Add a focused, recorded routine-workflow benchmark that pairs output cost with successful next-step
behavior:

  • snapshot -i and unchanged-snapshot suppression;
  • one mutation with --settle, including the unchanged-interactive tail;
  • get text or get attrs;
  • one actionable timeout/platform/selector failure that retains enough identity and session state
    for recovery;
  • CLI and MCP projections where they differ.

Track stable fixture-level UTF-8 bytes plus command count, fallback observation count, retry count,
and whether recovery preserves the current session. Keep live device measurements informational.

Add or extend small-model SkillGym/help-conformance cases so the outcome oracle verifies the next
command without a redundant observation. Do not make model-in-the-loop execution a required PR gate
unless the existing infrastructure is already deterministic and credential-free; the recorded
contract remains the fast CI gate.

Constraints

  • Do not optimize bytes by removing actionable refs, generation pins, stable error identity, retry
    guidance, or recovery handles.
  • Do not gate on live simulator output sizes.
  • Keep the work scoped to measurement and fixtures; response-default changes belong in a later,
    evidence-driven issue.

Acceptance criteria

  • A deterministic routine workflow reports output bytes and follow-up/retry counts.
  • Regression tests fail when the workflow requires a redundant observation or loses recovery
    fields.
  • At least one small-model task covers the same next-step decision.
  • pnpm test:output-economy, relevant benchmark tests, typecheck, lint, and Fallow pass.

Origin: post-#1174 architecture follow-up.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestin-progressWork is actively in progress

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions