feature: Add agent validation flow in foundry skill - #3113
Closed
Shawn Wang (shawnwang-w) wants to merge 1 commit into
Closed
feature: Add agent validation flow in foundry skill#3113Shawn Wang (shawnwang-w) wants to merge 1 commit into
Shawn Wang (shawnwang-w) wants to merge 1 commit into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new read-only “validate” workflow to the microsoft-foundry skill to support production-readiness audits of a single hosted agent, including a default ruleset and a standardized Markdown report template. This extends the Foundry skill’s routing to recognize validation/audit intents and introduces a corresponding integration stimulus.
Changes:
- Extend
microsoft-foundrySKILL routing/description to include a newvalidatesub-workflow and clarify dependency-setup exceptions for read-only validation. - Add a new validation workflow doc plus reference materials (default rules + report template).
- Add an integration stimulus intended to cover validation routing.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| plugins/azure-skills/skills/microsoft-foundry/SKILL.md | Adds validate as a first-class sub-workflow and updates routing/entry guidance. |
| plugins/azure-skills/skills/microsoft-foundry/foundry-agent/validate/validate.md | New read-only validation workflow instructions and constraints. |
| plugins/azure-skills/skills/microsoft-foundry/foundry-agent/validate/references/default-rules.md | Introduces the default production-readiness ruleset for repository-based reviews. |
| plugins/azure-skills/skills/microsoft-foundry/foundry-agent/validate/references/report-template.md | Adds a standardized Markdown report structure for validation output. |
| evals/azure-skills/microsoft-foundry/eval.yaml | Adds a new stimulus intended to test routing into the validation workflow. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+1137
to
+1150
| tags: | ||
| type: integration | ||
| tier: smoke | ||
| cost: llm | ||
| area: validation-routing | ||
| graders: | ||
| - type: skill-invocation | ||
| config: | ||
| required: | ||
| - microsoft-foundry | ||
| - type: completed | ||
| - type: output-not-matches | ||
| config: | ||
| pattern: "(?i)fatal error|unhandled exception|stack trace" |
anchenyi
reviewed
Aug 25, 2026
| @@ -1,6 +1,6 @@ | |||
| --- | |||
| name: microsoft-foundry | |||
| description: "Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and resources end to end with azd. USE FOR: azd ai agent, azd provision/deploy, hosted agent scaffold/develop/run/deploy, prompt agent create, create agent, update agent, add tool to agent, invoke agent, agent.yaml, evaluate agent, batch eval, continuous eval, continuous monitoring, agent CI/CD, optimize prompt, improve prompt, prompt optimizer, optimize agent instructions, Agent Optimizer scaffold, dataset curation from traces, deploy model, model fine-tuning (SFT/DPO/RFT), Foundry project, RBAC, role assignment, permissions, quota, capacity, region, troubleshoot agent, deployment failure, AI Services, create Foundry resource, knowledge index, customize deployment, onboard, availability, training-data, grader, distillation, fine-tuned model, large file upload. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure-prepare)." | |||
Collaborator
There was a problem hiding this comment.
Are those changes in the description of skill.md intented? Better not to change those trigger words
anchenyi
reviewed
Aug 25, 2026
| ### Dependency Check and Setup | ||
|
|
||
| **MANDATORY:** As the first step after this skill loads, run the dependency check and setup script below from this skill's root and wait for it to finish before continuing. The script checks first and installs only missing dependencies; it does not reinstall dependencies that are already available. | ||
| **MANDATORY:** As the first step after this skill loads, run the dependency check and setup script below from this skill's root and wait for it to finish before continuing. The script checks first and installs only missing dependencies; it does not reinstall dependencies that are already available. Skip this setup only for the read-only [validate](foundry-agent/validate/validate.md) workflow, which must not install dependencies. |
Collaborator
There was a problem hiding this comment.
This one only installs azd and related microsoft.foundry extension, I think it's fine to let all users do the dependency setup. Another thing is that, for telemetry purposes, this setup is critical for us to know the users count on skill load.
anchenyi
reviewed
Aug 25, 2026
| |-------------|------------------------| | ||
| | Create a new hosted agent end-to-end (scaffold + deploy + test) | [dependency check and setup](#dependency-check-and-setup) → [azd-guidance](foundry-agent/azd-guidance/azd-guidance.md) → [quick-start-hosted](foundry-agent/create/quick-start-hosted.md) (self-contained end-to-end) | | ||
| | Anything beyond the standard quickstart (existing code, migration, re-hosting, deployment customization, scaffold-time connections, A2A (Agent2Agent), recovery) | [dependency check and setup](#dependency-check-and-setup) → [azd-guidance](foundry-agent/azd-guidance/azd-guidance.md) → [create](foundry-agent/create/create-hosted.md) → [deploy](foundry-agent/deploy/deploy.md) → [invoke](foundry-agent/invoke/invoke.md) | | ||
| | Validate, audit, or review a hosted agent for production readiness | [validate](foundry-agent/validate/validate.md). This workflow is read-only; do not enter create, deploy, invoke, or observe unless the user separately requests that work. | |
Collaborator
There was a problem hiding this comment.
Better to add dependency check and setup → azd-guidance before the validate
anchenyi
reviewed
Aug 25, 2026
| - name: "Route hosted-agent validation" | ||
| prompt: "Validate my Microsoft Foundry hosted agent for production readiness." | ||
| tags: | ||
| type: integration |
Collaborator
There was a problem hiding this comment.
Integration test should not be under "Foundry E2E Checks". You can move it above.
anchenyi
reviewed
Aug 25, 2026
| the azd project root: | ||
|
|
||
| ```text | ||
| AZURE_DEV_USER_AGENT=microsoft_foundry_skill azd ai agent doctor --local-only --no-prompt |
Collaborator
There was a problem hiding this comment.
No need to add user agent here. azd-guidance has already covered this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a production-readiness validation flow for Microsoft Foundry hosted agents.
Checklist
cd tests && npm test)fix:,feat:,feature:,chore:,misc:,test:,eval:tests/,npm run test:vally -- --plugin <plugin-dirname> --skill <skill>)Related Issues
N/A