fix(studio): evaluate agents over chat completions

[marcusds]

Summary

Every evaluation Studio submitted targeted the agent proxy's /generate, which only NAT's FastAPI front end serves. A nemo-agents-spec-v1 agent is served by the Platform-owned Fabric server, which exposes /health, /v1/chat/completions and DELETE /v1/sessions/{id} and nothing else, so the request 404s and the run fails with nothing to show for it. Nothing in the submission path branched on config_format, so this was true for every Fabric agent.

This matters because nemo-agents-spec-v1 is the format nemo-build-agent produces by default, and since #1223 it is also what Studio's Create Example Agent produces — SAMPLE_AGENTS now holds a single Fabric entry. The documented way to build an agent, and Studio's own one-click path, both yield an agent that cannot be evaluated. This is broken on main today, not pending some future change.

After this change the target posts OpenAI chat completions for every agent, with no branch on config_format.

Related Issue

ASTD-410 (Linear).

Why one shape rather than a per-format branch

Both agent config formats already serve /v1/chat/completions:

• Fabric serves it natively (fabric/server.py).
• NAT serves it through its FastAPI front end, whose default workflow endpoint sets openai_api_v1_path="/v1/chat/completions" alongside the legacy legacy_path="/generate". No agent config in this repo overrides general.front_end, and both launchers — nat start fastapi in the subprocess backend and the container backend — take those defaults.

Studio already depends on this: the chat playground posts chat completions to every deployment via /-/v1, without regard to format, and the gateway carries an unknown-model patch written specifically for NAT's ChatResponse.

Only chat completions is common to both formats, so branching buys nothing and costs a lookup. Resolving config_format would mean fetching the agent entity before submit, gating submission on that fetch, and still choosing a wire format when the lookup is in flight or has failed — and the only safe-looking default, nat, is exactly the one that 404s for the agents this fixes. One shape removes the lookup, the fallback, and the race between them.

Changes

• agentEndpoint builds one target for both formats: POST { model, messages: [{ role: 'user', content: '{{ instruction }}' }], stream: false } to /-/v1/chat/completions, reading $.choices[0].message.content. buildAgentTarget and buildDatasetAgentTarget share it; the latter renders {{ prompt }}.
• DatasetEvalRowResultsPanel read the rendered prompt out of the logged request body at input_message, a key the new body does not have — it would have silently fallen back to dumping the raw dataset row. It now reads the last chat message, keeping input_message as a fallback so jobs submitted before this still render.
• Deletes AgentEvaluationsRoute/components/submitEvaluationSpec.ts, an unreferenced second copy of the submission logic still building the /generate target.
• Updates AgentEvaluationsRoute/AGENTS.md, which prescribed /generate as the eval target and told contributors to keep using it.
• Unit tests for both target builders and workspace-prefixed agent names, and a new DatasetEvalRowResultsPanel test covering the chat body, a multi-message transcript, the input_message fallback, and the raw-row fallback.

No evaluator or SDK change was required: the generic agent target already accepts a URL, a Jinja body template, and a response_path JSONPath.

Type of Change

• Code change (feature, bug fix, or refactor)
• Code change with documentation updates
• Documentation only
• Contributor tooling or automation
• CI, build, or test infrastructure

Quality Gates

• Tests added or updated for changed behavior
• Existing tests cover changed behavior — justification:
• Tests not applicable — justification:
• Documentation updated for user-visible behavior
• Documentation not applicable — justification: no user-facing documentation describes the eval target's wire shape. AgentEvaluationsRoute/AGENTS.md is contributor documentation and is updated here.

Verification

• Pull request title follows the repository's Conventional Commit format
• Every commit includes an appropriate Signed-off-by: trailer
• uv run pre-commit run -a passes, or any blocked checks are identified below
• Targeted tests pass, or tests are marked not applicable above
• No secrets, API keys, or credentials are included

Targeted validation, all at the current head after rebasing onto main:

• pnpm --filter nemo-studio-ui test — passed, 310 files / 2840 tests.
• pnpm --filter nemo-studio-ui typecheck — passed.
• pnpm --filter nemo-studio-ui lint — passed (--max-warnings 0).
• uv run pre-commit run -a — every hook passed except uv-lock, which failed on a local toolchain mismatch (uv.lock must be checked or updated with uv 0.9.14; this machine has 0.9.30). This PR changes no Python and no dependency metadata, and the separate Check for uv.lock drift hook passed. Not marked as passing above.
• Body templating checked against nemo_evaluator_sdk: render_template recurses through dicts and lists, so {{ instruction }} substitutes inside the nested messages entry rather than being sent literally.
• Response extraction checked against a real response proxied from a running agent, using jsonpath_ng.parse — the exact parser agent_inference.py imports, not the .ext variant. _extract_jsonpath returns matches[0].value, and $.choices[0].message.content returns the content.

Known limitation

A full evaluation job has not been run end to end. The transport, the body templating and the response extraction are each verified; that a job completes and scores is not.

Separately, every eval request opens a new Fabric session. Fabric starts a fresh runtime for each chat-completions call that carries no X-Nemo-Session-Id, the evaluator sends none, and sessions are reclaimed only by the 30-minute idle sweep — so a long task list leaves that many runtimes alive and pays a cold start per task. That is a platform-side concern rather than a Studio one; it is recorded in the route's AGENTS.md and addressed separately.

Summary by CodeRabbit

• New Features

  • Agent evaluations now use the non-streaming chat-completions endpoint for both task-based and dataset-based submissions.
  • Evaluation prompts support chat message formatting and extract responses from chat-completion results.

• Bug Fixes

  • Dataset evaluation results now display prompts from the latest chat message.
  • Legacy evaluation jobs using input_message remain supported.
  • Results fall back to the original row data when request details are unavailable.

• Documentation

  • Updated evaluation guidance for chat-completions usage, templating, response extraction, and session handling.

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	32002/40624	78.8%	63.5%
Integration Tests	18557/38550	48.1%	20.8%

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9baea652-d5bd-49aa-ae61-93fb73417c22

📥 Commits

Reviewing files that changed from the base of the PR and between 6d9163f and cfdd326.

📒 Files selected for processing (6)

• web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.test.tsx
• web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.tsx
• web/packages/studio/src/components/evaluation/submitEvaluationJob.test.ts
• web/packages/studio/src/components/evaluation/submitEvaluationJob.ts
• web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AGENTS.md
• web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts

💤 Files with no reviewable changes (1)

• web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts

---

📝 Walkthrough

Walkthrough

Evaluation targets now use non-streaming chat completions for task and dataset submissions. Dataset result panels support chat-completion messages and legacy inputs. Agent evaluation guidance and related specification code were updated.

Changes

Evaluation chat-completions migration

Layer / File(s)	Summary
Dataset row prompt extraction web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.tsx, web/packages/studio/src/components/evaluation/Jobs/datasetEval/DatasetEvalRowResultsPanel.test.tsx	Dataset rows now read the last chat message and fall back to input_message. Tests cover empty, current, multi-message, legacy, and missing-request cases.
Shared agent target construction web/packages/studio/src/components/evaluation/submitEvaluationJob.ts, web/packages/studio/src/components/evaluation/submitEvaluationJob.test.ts	Task and dataset targets now use /v1/chat/completions, render instruction or prompt in user messages, preserve stream: false, and extract $.choices[0].message.content.
Agent evaluation guidance and specification cleanup web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AGENTS.md, web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts	Documentation and examples now describe chat-completions targets, nested templates, and Fabric session cleanup. The former submitEvaluationSpec.ts module was removed.

Sequence Diagram(s)

sequenceDiagram
  participant EvaluationBuilder
  participant EvaluationRunner
  participant AgentChatCompletions
  EvaluationBuilder->>EvaluationRunner: render instruction or prompt as a user message
  EvaluationRunner->>AgentChatCompletions: send non-streaming chat-completions request
  AgentChatCompletions-->>EvaluationRunner: return choices[0].message.content

Loading

Suggested reviewers: briannewsom

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: updating Studio agent evaluations to use chat completions.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch astd-410-fabric-eval-endpoint/mschwab

---

_{Comment @coderabbitai help to get the list of available commands.}