fix: route "what did I complete?" to find-activity

[scottlovegrove]

Closes #564.

The bug

Asked "what did I actually get done last week?", Haiku 4.5 reaches for get-productivity-stats and find-completed-tasks and never gets to find-activity. Those answer different questions: find-activity reports completion events, so a daily recurring task completed five times shows five occurrences; find-completed-tasks lists completed tasks, and the recurring work silently vanishes from the answer.

The guidance already existed — on find-activity, and in the instructions block. Everywhere except the tool that actually gets picked. find-completed-tasks opened with "Get completed tasks", a perfect surface match for the question, and a model that has a hit stops reading. So the caveat now sits where the wrong choice is made, on both find-completed-tasks and get-productivity-stats.

The scenario could not measure it

Worth flagging, because it is the more interesting half. #564 reports Sonnet 5 at 0%, and Sonnet 5 turns out to answer the question perfectly. It opens with user-info — without today's date and the user's timezone, "last week" is not a date range — and then calls exactly what the scenario asks for:

user-info({}) -> find-activity({objectType:"task", eventType:"completed",
                 dateFrom:"2026-08-03T00:00:00+01:00", dateTo:"2026-08-10T00:00:00+01:00", ...})

The harness judged the first tool call outright, and user-info is not in the expect allowlist, so correct behaviour scored 0. This is the failure AGENTS.md:150 already warns about: an allowlist naming the tool under test cannot also name every reasonable lookup that precedes it. Fixing tool descriptions against that reading would have been optimising against noise.

Two changes to the harness, both narrow:

• A context tool (CONTEXT_TOOLS, currently just user-info) is answered with a canned result, and the next call is judged. Capped at MAX_CONTEXT_HOPS so a model that only ever gathers context still fails. A tool a scenario explicitly expects is never stubbed, so a future scenario testing the route to user-info stays measurable.
• A turn carrying several calls at once is judged on its first non-context call, not on whichever block came first — the order between "ask for the date" and "query the log" is arbitrary. A forbidden call outranks that, since catching it is the point of a forbid rule.

Adding user-info to expect would have been the cheap fix and the wrong one: the judge would pass on the lookup alone and check no arguments at all, scoring 100% while measuring nothing. AGENTS.md now documents both traps.

Filtering the answer to the asker

From review, and it turned out to be a real hole rather than a theoretical one. find-activity reports every user's events by default, so rerouting a first-person question to it means a shared project answers "what did I get done" with collaborators' completions mixed in. Asserting initiatorId in the scenario put it at 0% on both models — nothing was asking for it.

Sonnet 5 was already passing it unprompted and went to 100% once find-activity's description named it. Haiku needed the routing rule to name the order of the two calls — user-info first for the ID, then find-activity — because it was answering in one shot and had no ID to pass. Every Haiku run that calls user-info first passes; the ones that skip it are the remaining failures.

Numbers

completed-via-activity, 12 repeats, main vs this branch, both judged by the harness as it stands here (context hops, and initiatorId asserted):

model	main	this branch
claude-haiku-4-5	0%	67%
claude-sonnet-5	100%	100%

Sonnet's "before" is 100% because its routing was never broken — the harness fix alone is what surfaced that, and the argument assertions it already satisfied. Haiku's 67% is the honest 12-repeat figure; smaller samples ran higher, and the residual failure is always the same one, skipping user-info and so having no ID to filter by.

Splitting the two halves apart, since they moved for different reasons: on routing alone (which tool gets picked, the subject of #564), this branch is 100% on both models against main's 38% for Haiku. On filtering to the asker, 0% → 67% Haiku, 0% → 100% Sonnet.

Full sweep, 10 scenarios x 5 repeats: 100/100 on both models, up from 45/50 — completed-via-activity was the only failing scenario, and no other scenario moved.

npm run eval -- --label after --scenario completed-via-activity --repeats 12
npm run eval -- --label after-full --repeats 5

Cost

Token footprint 33,453 → 33,624 of a 35,000 budget: +138 across three tool descriptions and one input field, plus 33 on the instructions block (777 → 810). The budget is a ratchet, so to be explicit — this is a rise, bought with a routing failure fixed on the model that was getting it wrong, and a correctness gap closed on both.

Also from review

Both P3 notes applied: the user-info stub is now tied to UserInfoStructured with satisfies, so a renamed or added field on the real schema breaks the harness instead of drifting; and currentLocalTime now uses the toLocaleString('en-US', …) format the tool actually emits rather than ISO.

Left for later

• The instructions block bullet grew rather than shrank here, since naming the call order is what moved Haiku. Whether the tool descriptions can now carry that alone is worth a measured follow-up.
• The scenario's check does not assert objectType === "task", though the instructions and find-activity both ask for it. Both models set it anyway. Tightening it is a separate change, since it would move the baseline.

🤖 Generated with Claude Code

[doistbot]

This PR reroutes "what did I complete?" to find-activity by adding routing caveats to find-completed-tasks and get-productivity-stats, and upgrades the eval harness to stub context-tool lookups like user-info so the substantive call is judged instead of the preliminary one.

Few things worth tightening:

• find-activity defaults to all users, so in shared projects a first-person query can surface collaborators' completions — guide the model to fetch the current user ID via user-info and pass it as initiatorId, and add that assertion to the eval scenario.
• Use Object.hasOwn(CONTEXT_TOOLS, c.name) instead of in so a hallucinated name like __proto__ isn't treated as a context tool via prototype inheritance.
• Only bypass a context tool when it isn't explicitly expected by the scenario, otherwise a future expect: [ToolNames.USER_INFO] scenario can never pass — its expected call gets stubbed instead of judged.

I also included a few optional follow-up notes in the details below.

Optional follow-up notes (2)

• scripts/eval-instructions.ts:216: The user-info stub duplicates the UserInfoStructured shape by hand, and the comment above it notes the need to keep it in step with the real output schema. That link already exists: UserInfoStructured is exported from src/tools/user-info.ts. Annotating the entry with satisfies UserInfoStructured (keeping the outer map as Record<string, unknown> so future context tools with other shapes still fit) would turn the manual "keep in step" discipline into a compile-time check, so a renamed or newly-required field on the real schema breaks here instead of silently drifting.
• scripts/eval-instructions.ts:222: The currentLocalTime stub is '2026-08-11 09:30:00' (YYYY-MM-DD HH:MM:SS), but the real user-info tool formats it via toLocaleString('en-US', …), which produces '08/11/2026, 09:30:00' (MM/DD/YYYY, HH:MM:SS). The comment on CONTEXT_TOOLS says to keep the stub in step with the real output so the model reads it as it would a live one. The other date fields (weekStartDate, weekEndDate) already match the real getDateString format; currentLocalTime is the only one that doesn't. Low impact today (both models pass at 100%, and weekStartDate is the primary field for deriving "last week"), but worth aligning so a future model that leans on currentLocalTime isn't trained on a format it won't see in production.

_{Share Feedback • Review Logs}

[doist-release-bot]

🎉 This PR is included in version 12.5.6 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀