From 54b8704ecbba0ec8b2a56269a7bc4e7fbb1b753f Mon Sep 17 00:00:00 2001 From: Junhui Cheng Date: Sat, 20 Jun 2026 23:36:29 +0800 Subject: [PATCH 1/5] Update README setup instructions and remove obsolete files --- .env.example | 13 ------ AGENTS.md | 81 ++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 66 ------------------------------- README.md | 13 ++++++ deep_research_from_scratch | 1 + 5 files changed, 95 insertions(+), 79 deletions(-) delete mode 100644 .env.example create mode 100644 AGENTS.md delete mode 100644 CLAUDE.md create mode 160000 deep_research_from_scratch diff --git a/.env.example b/.env.example deleted file mode 100644 index 778272bf3..000000000 --- a/.env.example +++ /dev/null @@ -1,13 +0,0 @@ -OPENAI_API_KEY= -ANTHROPIC_API_KEY= -GOOGLE_API_KEY= -TAVILY_API_KEY= -LANGSMITH_API_KEY= -LANGSMITH_PROJECT= -LANGSMITH_TRACING= - -# Only necessary for Open Agent Platform -SUPABASE_KEY= -SUPABASE_URL= -# Should be set to true for a production deployment on Open Agent Platform. Should be set to false otherwise, such as for local development. -GET_API_KEYS_FROM_CONFIG=false \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..e1dc2115e --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,81 @@ +# Open Deep Research 仓库概览 + +## 项目描述 + +Open Deep Research 是一个可配置、完全开源的深度研究智能体(deep research agent)，支持多个模型提供商、搜索工具以及 MCP（Model Context Protocol，模型上下文协议）服务器。它能够通过并行处理自动执行研究任务，并生成内容全面的研究报告。 + +## 仓库结构 + +### 根目录 + +* `README.md` - 完整的项目文档，包含快速入门指南 +* `pyproject.toml` - Python 项目配置与依赖声明 +* `langgraph.json` - LangGraph 配置文件，用于定义主图(Graph)的入口点 +* `uv.lock` - UV 包管理器的依赖锁定文件 +* `LICENSE` - MIT 许可证 +* `.env.example` - 环境变量模板（不受版本控制追踪） + +### 核心实现（`src/open_deep_research/`） + +* `deep_researcher.py` - LangGraph 的主要实现文件（入口点：`deep_researcher`） +* `configuration.py` - 配置管理与设置 +* `state.py` - 图状态(Graph state)定义与数据结构 +* `prompts.py` - 系统提示词与提示词模板 +* `utils.py` - 工具函数与辅助功能 +* `files/` - 研究输出文件与示例文件 + +### 遗留实现（`src/legacy/`） + +包含两种早期的研究实现： + +* `graph.py` - 带有人机协同(human-in-the-loop)机制的规划与执行(plan-and-execute)工作流 +* `multi_agent.py` - 监督者—研究员(supervisor-researcher)多智能体架构 +* `legacy.md` - 遗留实现的说明文档 +* `CLAUDE.md` - 面向遗留实现的 Claude 专用指令 +* `tests/` - 遗留实现专用测试 + +### 安全模块（`src/security/`） + +* `auth.py` - 用于 LangGraph 部署的身份验证处理程序 + +### 测试（`tests/`） + +* `run_evaluate.py` - 主评估脚本，配置为在 Deep Research Bench 上运行 +* `evaluators.py` - 专用评估函数 +* `prompts.py` - 评估提示词与评估标准 +* `pairwise_evaluation.py` - 对比评估工具 +* `supervisor_parallel_evaluation.py` - 多线程并行评估 + +### 示例（`examples/`） + +* `arxiv.md` - ArXiv 研究示例 +* `pubmed.md` - PubMed 研究示例 +* `inference-market.md` - 推理市场分析示例 + +### 手把手教学（deep_research_from_scratch/） +* 改文件夹下放的该项目进面向人类初学者的教学jupyter notebook内容，可视为与本项目的实际运行和使用完全解耦的模块 + +## 核心技术 + +* **LangGraph** - 工作流编排与图执行(Graph execution) +* **LangChain** - 大语言模型集成与工具调用(tool calling) +* **多个 LLM 提供商** - 支持 OpenAI、Anthropic、Google、Groq 和 DeepSeek +* **搜索 API** - 支持 Tavily、OpenAI/Anthropic 原生搜索、DuckDuckGo 和 Exa +* **MCP Servers** - 通过 Model Context Protocol 扩展智能体能力 + +## 开发命令 + +* `uvx langgraph dev` - 启动带有 LangGraph Studio 的开发服务器 +* `python tests/run_evaluate.py` - 运行完整评估 +* `ruff check` - 执行代码 lint 检查 +* `mypy` - 执行类型检查 + +## 配置 + +所有设置均可通过以下方式进行配置： + +* 环境变量（`.env` 文件） +* LangGraph Studio 中的 Web UI +* 直接修改配置文件 + +关键设置包括模型选择、搜索 API 选择、并发限制以及 MCP server 配置。 diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index d0bf2abb5..000000000 --- a/CLAUDE.md +++ /dev/null @@ -1,66 +0,0 @@ -# Open Deep Research Repository Overview - -## Project Description -Open Deep Research is a configurable, fully open-source deep research agent that works across multiple model providers, search tools, and MCP (Model Context Protocol) servers. It enables automated research with parallel processing and comprehensive report generation. - -## Repository Structure - -### Root Directory -- `README.md` - Comprehensive project documentation with quickstart guide -- `pyproject.toml` - Python project configuration and dependencies -- `langgraph.json` - LangGraph configuration defining the main graph entry point -- `uv.lock` - UV package manager lock file -- `LICENSE` - MIT license -- `.env.example` - Environment variables template (not tracked) - -### Core Implementation (`src/open_deep_research/`) -- `deep_researcher.py` - Main LangGraph implementation (entry point: `deep_researcher`) -- `configuration.py` - Configuration management and settings -- `state.py` - Graph state definitions and data structures -- `prompts.py` - System prompts and prompt templates -- `utils.py` - Utility functions and helpers -- `files/` - Research output and example files - -### Legacy Implementations (`src/legacy/`) -Contains two earlier research implementations: -- `graph.py` - Plan-and-execute workflow with human-in-the-loop -- `multi_agent.py` - Supervisor-researcher multi-agent architecture -- `legacy.md` - Documentation for legacy implementations -- `CLAUDE.md` - Legacy-specific Claude instructions -- `tests/` - Legacy-specific tests - -### Security (`src/security/`) -- `auth.py` - Authentication handler for LangGraph deployment - -### Testing (`tests/`) -- `run_evaluate.py` - Main evaluation script configured to run on deep research bench -- `evaluators.py` - Specialized evaluation functions -- `prompts.py` - Evaluation prompts and criteria -- `pairwise_evaluation.py` - Comparative evaluation tools -- `supervisor_parallel_evaluation.py` - Multi-threaded evaluation - -### Examples (`examples/`) -- `arxiv.md` - ArXiv research example -- `pubmed.md` - PubMed research example -- `inference-market.md` - Inference market analysis examples - -## Key Technologies -- **LangGraph** - Workflow orchestration and graph execution -- **LangChain** - LLM integration and tool calling -- **Multiple LLM Providers** - OpenAI, Anthropic, Google, Groq, DeepSeek support -- **Search APIs** - Tavily, OpenAI/Anthropic native search, DuckDuckGo, Exa -- **MCP Servers** - Model Context Protocol for extended capabilities - -## Development Commands -- `uvx langgraph dev` - Start development server with LangGraph Studio -- `python tests/run_evaluate.py` - Run comprehensive evaluations -- `ruff check` - Code linting -- `mypy` - Type checking - -## Configuration -All settings configurable via: -- Environment variables (`.env` file) -- Web UI in LangGraph Studio -- Direct configuration modification - -Key settings include model selection, search API choice, concurrency limits, and MCP server configurations. \ No newline at end of file diff --git a/README.md b/README.md index 5bfa38ac5..256e0adbe 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,15 @@ uv venv source .venv/bin/activate # On Windows: .venv\Scripts\activate ``` +```bash +git clone https://github.com/langchain-ai/open_deep_research.git +cd open_deep_research + +conda create -n open-deep-research python=3.11 -y +conda activate open-deep-research +python -m pip install --upgrade pip setuptools wheel +``` + 2. Install dependencies: ```bash uv sync @@ -35,6 +44,10 @@ uv sync uv pip install -r pyproject.toml ``` +```bash +pip install -e . # 可编辑模式安装当前目录中的项目，并基于pyproject.toml进行依赖解析 +``` + 3. Set up your `.env` file to customize the environment variables (for model selection, search tools, and other configuration settings): ```bash cp .env.example .env diff --git a/deep_research_from_scratch b/deep_research_from_scratch new file mode 160000 index 000000000..b58bf73e6 --- /dev/null +++ b/deep_research_from_scratch @@ -0,0 +1 @@ +Subproject commit b58bf73e66868b10faee1ea92acd00f2af42d75b From 7e54c115035f8a7a1ae041300384ea354724d290 Mon Sep 17 00:00:00 2001 From: magic-ZSS <1165626853@qq.com> Date: Sun, 21 Jun 2026 02:15:01 +0800 Subject: [PATCH 2/5] Refactor deep research workflow and evaluation tooling --- .agents/skills/harness-creator/README.md | 83 ++++++ .agents/skills/harness-creator/SKILL.md | 107 +++++++ .agents/skills/harness-creator/SKILL.md.en | 107 +++++++ .../skills/harness-creator/agents/openai.yaml | 7 + .../skills/harness-creator/evals/evals.json | 136 +++++++++ .../references/context-engineering-pattern.md | 150 ++++++++++ .../harness-creator/references/gotchas.md | 209 ++++++++++++++ .../references/lifecycle-bootstrap-pattern.md | 264 ++++++++++++++++++ .../references/memory-persistence-pattern.md | 110 ++++++++ .../references/multi-agent-pattern.md | 193 +++++++++++++ .../references/skill-runtime-pattern.md | 43 +++ .../references/tool-registry-pattern.md | 200 +++++++++++++ .../scripts/create-harness.mjs | 75 +++++ .../scripts/render-assessment-html.mjs | 26 ++ .../harness-creator/scripts/run-benchmark.mjs | 121 ++++++++ .../scripts/validate-harness.mjs | 43 +++ .../harness-creator/templates/agents.md | 68 +++++ .../templates/feature-list.json | 44 +++ .../templates/feature-list.schema.json | 47 ++++ .../skills/harness-creator/templates/init.sh | 75 +++++ .../harness-creator/templates/progress.md | 51 ++++ .../templates/session-handoff.md | 40 +++ AGENTS.md | 97 +++---- AGENTS.override.md | 81 ++++++ README.md | 14 +- feature_list.json | 12 + init.sh | 19 ++ progress.md | 52 ++++ session-handoff.md | 45 +++ skills-lock.json | 11 + 30 files changed, 2458 insertions(+), 72 deletions(-) create mode 100644 .agents/skills/harness-creator/README.md create mode 100644 .agents/skills/harness-creator/SKILL.md create mode 100644 .agents/skills/harness-creator/SKILL.md.en create mode 100644 .agents/skills/harness-creator/agents/openai.yaml create mode 100644 .agents/skills/harness-creator/evals/evals.json create mode 100644 .agents/skills/harness-creator/references/context-engineering-pattern.md create mode 100644 .agents/skills/harness-creator/references/gotchas.md create mode 100644 .agents/skills/harness-creator/references/lifecycle-bootstrap-pattern.md create mode 100644 .agents/skills/harness-creator/references/memory-persistence-pattern.md create mode 100644 .agents/skills/harness-creator/references/multi-agent-pattern.md create mode 100644 .agents/skills/harness-creator/references/skill-runtime-pattern.md create mode 100644 .agents/skills/harness-creator/references/tool-registry-pattern.md create mode 100644 .agents/skills/harness-creator/scripts/create-harness.mjs create mode 100644 .agents/skills/harness-creator/scripts/render-assessment-html.mjs create mode 100644 .agents/skills/harness-creator/scripts/run-benchmark.mjs create mode 100644 .agents/skills/harness-creator/scripts/validate-harness.mjs create mode 100644 .agents/skills/harness-creator/templates/agents.md create mode 100644 .agents/skills/harness-creator/templates/feature-list.json create mode 100644 .agents/skills/harness-creator/templates/feature-list.schema.json create mode 100644 .agents/skills/harness-creator/templates/init.sh create mode 100644 .agents/skills/harness-creator/templates/progress.md create mode 100644 .agents/skills/harness-creator/templates/session-handoff.md create mode 100644 AGENTS.override.md create mode 100644 feature_list.json create mode 100644 init.sh create mode 100644 progress.md create mode 100644 session-handoff.md create mode 100644 skills-lock.json diff --git a/.agents/skills/harness-creator/README.md b/.agents/skills/harness-creator/README.md new file mode 100644 index 000000000..862766338 --- /dev/null +++ b/.agents/skills/harness-creator/README.md @@ -0,0 +1,83 @@ +# harness-creator + +A compact skill for building and auditing harnesses around AI coding agents. + +It helps a repository provide five things agents need: instructions, state, verification, scope boundaries, and lifecycle handoff. + +## Install + +```bash +npx skills add walkinglabs/learn-harness-engineering --skill harness-creator +``` + +Or copy `skills/harness-creator/` into your skill path. + +## Use + +```bash +node skills/harness-creator/scripts/create-harness.mjs --target /path/to/project +node skills/harness-creator/scripts/validate-harness.mjs --target /path/to/project +node skills/harness-creator/scripts/run-benchmark.mjs --target /path/to/project --html /path/to/report.html +``` + +The scripts use only Node.js built-in modules. They can be run after copying the skill directory into another repository. + +## What It Creates + +- `AGENTS.md` or `CLAUDE.md` +- `feature_list.json` +- `progress.md` +- `init.sh` +- `session-handoff.md` + +`create-harness.mjs` detects common project types and package managers. It supports Node/npm/pnpm/yarn/bun, Python, Go, Rust, Maven, Gradle, and .NET at a basic verification-command level. + +## What It Checks + +`validate-harness.mjs` scores the five harness subsystems: + +1. Instructions +2. State +3. Verification +4. Scope +5. Lifecycle + +The score is structural. It tells you whether the harness is present and coherent; it does not replace real before/after agent-session testing. + +## Status + +- [x] Minimal harness scaffolding +- [x] Five-subsystem validation +- [x] HTML assessment report +- [x] Structural benchmark report +- [x] 10 eval cases +- [x] Generic verification detection for common stacks +- [ ] Optional real before/after agent-session replay + +## Files + +```text +harness-creator/ +├── SKILL.md +├── metadata.json +├── agents/openai.yaml +├── scripts/ +│ ├── create-harness.mjs +│ ├── validate-harness.mjs +│ ├── render-assessment-html.mjs +│ ├── run-benchmark.mjs +│ └── lib/harness-utils.mjs +├── templates/ +│ ├── agents.md +│ ├── feature-list.json +│ ├── feature-list.schema.json +│ ├── init.sh +│ ├── progress.md +│ └── session-handoff.md +├── references/ +└── evals/evals.json +``` + +## Boundaries + +This skill is for harness engineering, not model selection, prompt tuning alone, or app architecture. Keep project-specific facts in the target repository. diff --git a/.agents/skills/harness-creator/SKILL.md b/.agents/skills/harness-creator/SKILL.md new file mode 100644 index 000000000..802fed82a --- /dev/null +++ b/.agents/skills/harness-creator/SKILL.md @@ -0,0 +1,107 @@ +--- +name: harness-creator +description: >- + Build, audit, and improve lightweight harnesses for AI coding agents: AGENTS.md/CLAUDE.md, + feature state, verification workflows, scope boundaries, lifecycle handoff, + memory persistence, context control, tool safety, and multi-agent coordination. +license: MIT +--- + +# Harness Creator + +Use this skill to make a repository easier for coding agents to start, stay in scope, verify work, and resume across sessions. Keep the harness small enough that agents actually follow it. + +Not for model selection, prompt tuning in isolation, chat UI design, or general app architecture. + +## Core Model + +Every useful coding-agent harness has five subsystems: + +| Subsystem | Minimal artifact | Purpose | +|---|---|---| +| Instructions | `AGENTS.md` or `CLAUDE.md` | Startup path, working rules, definition of done | +| State | `feature_list.json`, `progress.md` | Current feature, status, evidence, next step | +| Verification | `init.sh` or documented commands | Tests/checks the agent must run before claiming done | +| Scope | Feature dependencies and done criteria | Prevents overreach and half-finished work | +| Lifecycle | `session-handoff.md`, end-of-session routine | Makes the next session restartable | + +## First Move + +1. Inspect what already exists: instruction files, feature/state files, verification commands, docs, package manifests. +2. Ask only for missing context that cannot be inferred safely: target agent, desired file name, tolerance for structure, and whether overwriting is allowed. +3. Prefer a minimal harness first. Add memory, tool safety, multi-agent, or benchmark details only when the user's problem calls for them. + +## Common Tasks + +### Create a harness + +Use the bundled script when working on a local repository: + +```bash +node skills/harness-creator/scripts/create-harness.mjs --target /path/to/project +``` + +Options: + +- `--agent-file CLAUDE.md` for Claude-oriented projects. +- `--package-manager npm|pnpm|yarn|bun` when detection is wrong. +- `--commands "cmd one,cmd two"` for custom verification. +- `--force` only after confirming overwrites are acceptable. + +Then explain what was created and how the user should replace placeholder feature entries. + +### Audit an existing harness + +Run: + +```bash +node skills/harness-creator/scripts/validate-harness.mjs --target /path/to/project +``` + +Report the five subsystem scores, the lowest-scoring area, and the first 2-3 changes that would improve reliability. Treat the lowest score as a candidate bottleneck; confirm with failures, logs, or task outcomes before claiming causality. + +### Produce a report + +Use when the user wants a shareable assessment: + +```bash +node skills/harness-creator/scripts/render-assessment-html.mjs --target /path/to/project +node skills/harness-creator/scripts/run-benchmark.mjs --target /path/to/project --html /path/to/report.html +``` + +Be clear that this is a structural benchmark. Real effectiveness still needs before/after agent sessions on representative tasks. + +## When to Read References + +Load only the reference needed for the user's problem: + +- Memory across sessions: [Memory Persistence](references/memory-persistence-pattern.md) +- Reusable workflows as skills: [Skill Runtime](references/skill-runtime-pattern.md) +- Permissions, tools, concurrency: [Tool Registry & Safety](references/tool-registry-pattern.md) +- Context budget and progressive disclosure: [Context Engineering](references/context-engineering-pattern.md) +- Delegation and parallel agents: [Multi-Agent Coordination](references/multi-agent-pattern.md) +- Hooks, startup, long-running work: [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) +- Non-obvious failure modes: [Gotchas](references/gotchas.md) + +## Design Rules + +- Keep the root instruction file short: routing and invariants, not a full manual. +- Put project facts in project docs, not in the skill. +- Make verification commands explicit and runnable. +- Require evidence before marking a feature done. +- Use one active feature unless the harness has explicit multi-agent ownership boundaries. +- Prefer append/update state files over relying on chat history. +- Never hide destructive behavior in scripts; overwrites require explicit user approval. + +## Deliverable Checklist + +For a usable minimal harness, leave the target project with: + +- [ ] `AGENTS.md` or `CLAUDE.md` +- [ ] `feature_list.json` +- [ ] `progress.md` +- [ ] `init.sh` +- [ ] Optional `session-handoff.md` for multi-session work +- [ ] Documented verification evidence or next action + +If you cannot create files, provide exact file contents and commands instead. diff --git a/.agents/skills/harness-creator/SKILL.md.en b/.agents/skills/harness-creator/SKILL.md.en new file mode 100644 index 000000000..802fed82a --- /dev/null +++ b/.agents/skills/harness-creator/SKILL.md.en @@ -0,0 +1,107 @@ +--- +name: harness-creator +description: >- + Build, audit, and improve lightweight harnesses for AI coding agents: AGENTS.md/CLAUDE.md, + feature state, verification workflows, scope boundaries, lifecycle handoff, + memory persistence, context control, tool safety, and multi-agent coordination. +license: MIT +--- + +# Harness Creator + +Use this skill to make a repository easier for coding agents to start, stay in scope, verify work, and resume across sessions. Keep the harness small enough that agents actually follow it. + +Not for model selection, prompt tuning in isolation, chat UI design, or general app architecture. + +## Core Model + +Every useful coding-agent harness has five subsystems: + +| Subsystem | Minimal artifact | Purpose | +|---|---|---| +| Instructions | `AGENTS.md` or `CLAUDE.md` | Startup path, working rules, definition of done | +| State | `feature_list.json`, `progress.md` | Current feature, status, evidence, next step | +| Verification | `init.sh` or documented commands | Tests/checks the agent must run before claiming done | +| Scope | Feature dependencies and done criteria | Prevents overreach and half-finished work | +| Lifecycle | `session-handoff.md`, end-of-session routine | Makes the next session restartable | + +## First Move + +1. Inspect what already exists: instruction files, feature/state files, verification commands, docs, package manifests. +2. Ask only for missing context that cannot be inferred safely: target agent, desired file name, tolerance for structure, and whether overwriting is allowed. +3. Prefer a minimal harness first. Add memory, tool safety, multi-agent, or benchmark details only when the user's problem calls for them. + +## Common Tasks + +### Create a harness + +Use the bundled script when working on a local repository: + +```bash +node skills/harness-creator/scripts/create-harness.mjs --target /path/to/project +``` + +Options: + +- `--agent-file CLAUDE.md` for Claude-oriented projects. +- `--package-manager npm|pnpm|yarn|bun` when detection is wrong. +- `--commands "cmd one,cmd two"` for custom verification. +- `--force` only after confirming overwrites are acceptable. + +Then explain what was created and how the user should replace placeholder feature entries. + +### Audit an existing harness + +Run: + +```bash +node skills/harness-creator/scripts/validate-harness.mjs --target /path/to/project +``` + +Report the five subsystem scores, the lowest-scoring area, and the first 2-3 changes that would improve reliability. Treat the lowest score as a candidate bottleneck; confirm with failures, logs, or task outcomes before claiming causality. + +### Produce a report + +Use when the user wants a shareable assessment: + +```bash +node skills/harness-creator/scripts/render-assessment-html.mjs --target /path/to/project +node skills/harness-creator/scripts/run-benchmark.mjs --target /path/to/project --html /path/to/report.html +``` + +Be clear that this is a structural benchmark. Real effectiveness still needs before/after agent sessions on representative tasks. + +## When to Read References + +Load only the reference needed for the user's problem: + +- Memory across sessions: [Memory Persistence](references/memory-persistence-pattern.md) +- Reusable workflows as skills: [Skill Runtime](references/skill-runtime-pattern.md) +- Permissions, tools, concurrency: [Tool Registry & Safety](references/tool-registry-pattern.md) +- Context budget and progressive disclosure: [Context Engineering](references/context-engineering-pattern.md) +- Delegation and parallel agents: [Multi-Agent Coordination](references/multi-agent-pattern.md) +- Hooks, startup, long-running work: [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) +- Non-obvious failure modes: [Gotchas](references/gotchas.md) + +## Design Rules + +- Keep the root instruction file short: routing and invariants, not a full manual. +- Put project facts in project docs, not in the skill. +- Make verification commands explicit and runnable. +- Require evidence before marking a feature done. +- Use one active feature unless the harness has explicit multi-agent ownership boundaries. +- Prefer append/update state files over relying on chat history. +- Never hide destructive behavior in scripts; overwrites require explicit user approval. + +## Deliverable Checklist + +For a usable minimal harness, leave the target project with: + +- [ ] `AGENTS.md` or `CLAUDE.md` +- [ ] `feature_list.json` +- [ ] `progress.md` +- [ ] `init.sh` +- [ ] Optional `session-handoff.md` for multi-session work +- [ ] Documented verification evidence or next action + +If you cannot create files, provide exact file contents and commands instead. diff --git a/.agents/skills/harness-creator/agents/openai.yaml b/.agents/skills/harness-creator/agents/openai.yaml new file mode 100644 index 000000000..500d7ae09 --- /dev/null +++ b/.agents/skills/harness-creator/agents/openai.yaml @@ -0,0 +1,7 @@ +interface: + display_name: "Harness Creator" + short_description: "Scaffold and audit coding-agent harnesses" + default_prompt: "Use $harness-creator to create, validate, and benchmark a production-ready coding-agent harness for this repository." + +policy: + allow_implicit_invocation: true diff --git a/.agents/skills/harness-creator/evals/evals.json b/.agents/skills/harness-creator/evals/evals.json new file mode 100644 index 000000000..723d5ee68 --- /dev/null +++ b/.agents/skills/harness-creator/evals/evals.json @@ -0,0 +1,136 @@ +{ + "skill_name": "harness-creator", + "evals": [ + { + "id": 1, + "name": "Minimal Harness Creation", + "prompt": "I have a new TypeScript + React project with no agent setup. Create a minimal harness that makes my agent reliable for single-feature development.", + "expected_output": "AGENTS.md (~50-100 lines), feature_list.json with 3-5 placeholder features, init.sh with verification commands", + "files": [], + "expectations": [ + "AGENTS.md includes startup workflow (read files, run init, check feature list)", + "AGENTS.md includes one-feature-at-a-time policy", + "feature_list.json has valid JSON with id, name, description, status fields", + "init.sh runs install, type/static check, test, and build commands when available", + "All files are in project root directory" + ] + }, + { + "id": 2, + "name": "Session Continuity Setup", + "prompt": "My agent forgets everything between sessions. I need persistent memory and session handoff so it can work on multi-day features.", + "expected_output": "progress.md template, session-handoff.md structure, memory directory setup instructions", + "files": [], + "expectations": [ + "progress.md includes sections: current state, what's done, what's in progress, blockers, next session should", + "session-handoff.md includes: what was accomplished, what remains, blockers/decisions, files modified", + "Instructions for memory directory creation (.claude/memory/ or similar)", + "Two-step save invariant explained (topic file then index)" + ] + }, + { + "id": 3, + "name": "Harness Assessment", + "prompt": "I have an existing AGENTS.md file but my agent still breaks things. Assess my harness and tell me what to improve first.", + "expected_output": "Five-subsystem assessment with scores 1-5 for each, bottleneck identified, prioritized improvement plan", + "files": ["existing-AGENTS.md"], + "expectations": [ + "Assessment covers all 5 subsystems: Instructions, State, Verification, Scope, Lifecycle", + "Each subsystem scored 1-5 with justification", + "Lowest-scoring subsystem identified as bottleneck or candidate bottleneck", + "Prioritized improvement plan with 2-3 concrete next steps" + ] + }, + { + "id": 4, + "name": "Verification Workflow Design", + "prompt": "My agent says 'done' but the tests fail. Design a verification workflow that forces the agent to actually verify before claiming completion.", + "expected_output": "Verification commands list, AGENTS.md section updates, optional quality score tracking", + "files": [], + "expectations": [ + "Explicit verification commands listed (tests, lint, type-check, build)", + "AGENTS.md includes Definition of Done section with verification requirement", + "End-of-session checklist includes verification evidence recording", + "Failure path says not to claim done when verification fails" + ] + }, + { + "id": 5, + "name": "Memory Taxonomy Design", + "prompt": "I want my agent to remember project conventions and user preferences across sessions. Design a memory taxonomy and tell me what belongs where.", + "expected_output": "Memory layer definitions, type taxonomy for auto-memory, what to save vs what to skip", + "files": [], + "expectations": [ + "Instruction memory defined (human-curated, version-controlled)", + "Auto-memory defined (agent-written, persistent)", + "Type taxonomy with 3-4 types (e.g., user/feedback/project/reference)", + "Clear guidance on what NOT to save (derivable content)" + ] + }, + { + "id": 6, + "name": "Tool Safety Design", + "prompt": "My custom coding agent can run shell commands and edit files. Design the tool registry and permission harness so it is powerful but does not silently do dangerous things.", + "expected_output": "Tool registry policy with per-call concurrency classification, default permissions, denial handling, and audit trail", + "files": [], + "expectations": [ + "Sensitive tools default to ask or deny instead of allow", + "Concurrency safety is classified per call rather than per tool", + "Permission evaluation side effects are acknowledged and not cached unsafely", + "Audit trail records command, decision, and reason" + ] + }, + { + "id": 7, + "name": "Context Budget Plan", + "prompt": "My agent loads too many docs at startup and gets slow. Design a context strategy that keeps important context while controlling token cost.", + "expected_output": "Progressive disclosure plan using select/write/compress/isolate operations and explicit context budgets", + "files": [], + "expectations": [ + "Defines always-loaded metadata versus on-demand references", + "Uses SELECT, WRITE, COMPRESS, and ISOLATE operations", + "Includes hard caps or budget thresholds", + "Explains invalidation for memoized context builders" + ] + }, + { + "id": 8, + "name": "Multi-Agent Coordination", + "prompt": "I want multiple agents to work on a large refactor without stepping on each other. Design a multi-agent harness and rules for delegation.", + "expected_output": "Coordinator/delegation design with ownership boundaries, context sharing policy, and merge/review gates", + "files": [], + "expectations": [ + "Defines coordinator, worker, and optional reviewer responsibilities", + "Assigns disjoint file or module ownership", + "Prevents recursive fork children from forking again", + "Includes integration and verification gate before claiming done" + ] + }, + { + "id": 9, + "name": "Lifecycle Bootstrap", + "prompt": "Every new session starts inconsistently. Design startup, hook, and handoff lifecycle rules so a new agent can resume safely.", + "expected_output": "Lifecycle bootstrap design with init.sh, clean-state checks, handoff reads, and hook trust boundaries", + "files": [], + "expectations": [ + "init.sh is the standard startup and verification entrypoint", + "Startup reads AGENTS.md, feature_list.json, progress.md, and handoff when present", + "End-of-session procedure records evidence, blockers, and next step", + "Hooks are gated by trust and failure behavior is explicit" + ] + }, + { + "id": 10, + "name": "Scripted Harness Validation", + "prompt": "Use the harness-creator scripts to scaffold a harness, validate it, and produce an HTML assessment report for a repository.", + "expected_output": "Commands using create-harness.mjs, validate-harness.mjs, and render-assessment-html.mjs plus interpretation of score", + "files": [], + "expectations": [ + "Uses create-harness.mjs with a target directory", + "Uses validate-harness.mjs and explains five-subsystem score output", + "Uses render-assessment-html.mjs or run-benchmark.mjs with --html", + "Explains that structural benchmark complements but does not replace real before/after agent sessions" + ] + } + ] +} diff --git a/.agents/skills/harness-creator/references/context-engineering-pattern.md b/.agents/skills/harness-creator/references/context-engineering-pattern.md new file mode 100644 index 000000000..6bf6e3d83 --- /dev/null +++ b/.agents/skills/harness-creator/references/context-engineering-pattern.md @@ -0,0 +1,150 @@ +# Context Engineering Pattern + +## Problem + +Agents fail when context is managed poorly: +- **Too much context** → Session startup is slow, token costs explode, model gets lost in details +- **Too little context** → Agent makes wrong assumptions, reinvents wheels, violates conventions +- **Wrong context** → Agent focuses on low-level details, misses architectural constraints + +Context is not a dump. It's a budget that must be managed with explicit operations. + +## Golden Rules + +### Four Context Operations + +Every token in the window should earn its place through one of four operations: + +1. **SELECT** — Load context just-in-time, not all-at-once +2. **WRITE** — Agent writes back to persistent storage (memory, state, rules) +3. **COMPRESS** — Reactive compaction of older turns mid-session +4. **ISOLATE** — Delegated work must not pollute parent context + +### Progressive Disclosure + +Three-tier loading: + +``` +Tier 1: Metadata (always present, cheap) + → Feature list, memory index, session status + +Tier 2: Instructions (loaded on activation) + → AGENTS.md, skill bodies, style guides + +Tier 3: Resources (loaded on demand) + → Architecture docs, API references, examples +``` + +### Memoize Expensive Builders, Invalidate Explicitly + +Context builders (e.g., "load all recent git commits") should be memoized to avoid redundant work, but **must** be invalidated at known mutation points — not reactively. Every mutation point must clear its corresponding cache. + +## When To Use + +- Agent performance degrades in long sessions +- Startup is slow due to eager context loading +- Delegated work pollutes the parent context +- Token costs are unpredictable + +## Tradeoffs + +| Decision | Benefit | Cost | +|---|---|---| +| JIT loading | Fast startup, low idle cost | Agent can't reason about skills until activated | +| Hard caps per block | Predictable token budget | May truncate useful context | +| Manual cache invalidation | No reactive staleness | Developer must add invalidation at each mutation | +| Isolation for delegation | Clean parent context | Child can't see parent's accumulated context | + +## Implementation Patterns + +### Select Pattern + +```markdown +## Startup Context (Loaded Immediately) + +- Repository root path +- Tech stack (one line) +- Active feature ID from feature_list.json + +## On-Demand Context (Loaded When Triggered) + +- Skill: Read when skill activates +- Architecture docs: Read when implementing new feature +- API reference: Read when calling external services +``` + +**Key moves:** +- Audit current context cost per turn +- Apply hard caps to every variable-length block +- Add truncation recovery pointers ("call list_files for full output") + +### Compress Pattern + +Long sessions exhaust the window. Reactive compaction: + +1. **Trigger**: Context usage exceeds threshold (e.g., 80%) +2. **Summarize**: Older turns (first 50% by token count) +3. **Preserve**: Recent context (last 20% of turns) +4. **Label**: Mark snapshot as "compacted at turn N" + +```markdown +## Session Summary (Turns 1-15, compacted) + +**Goal**: Implement Q&A feature with citations +**Decisions made**: +- Use streaming response for UX +- Citation format: [doc:chunk] inline references +**Key files created**: +- src/services/QaService.ts +- src/shared/types.ts (extended with QaResult) +``` + +### Isolate Pattern + +Delegated work must not pollute parent context: + +| Pattern | Context Sharing | Best For | +|---|---|---| +| **Coordinator** (zero inheritance) | None — workers start fresh | Complex multi-phase tasks | +| **Fork** (full inheritance) | Full — single-level only | Quick parallel splits | +| **Swarm** (peer-to-peer) | Shared task list | Long-running independent work | + +**Key constraint**: Fork is single-level only — recursive forks multiply context cost exponentially. + +## Gotchas + +1. **Most async work skips "pending" state** — work units register directly as "running" +2. **Context builders are memoized but manually invalidated** — add invalidation or face staleness +3. **Truncation is silent until it fires** — hard caps enforced at read time +4. **Isolation boundary must be enforced at call time** — don't just remove tools from prompt + +## Related Patterns + +- [Memory Persistence](memory-persistence-pattern.md) — How memory layers interact with context +- [Multi-agent Coordination](multi-agent-pattern.md) — Context sharing across agents + +## Template: Context Budget + +```markdown +## Context Budget (Session) + +| Category | Budget | Current | Status | +|----------|--------|---------|--------| +| System prompt | 2,000 | 1,850 | ✓ | +| Instruction files | 3,000 | 2,400 | ✓ | +| Memory index | 1,000 | 600 | ✓ | +| Session history | 10,000 | 4,200 | ✓ | +| Working context | 15,000 | 3,100 | ✓ | +| **Total** | **31,000** | **12,150** | 39% used | + +**Compaction trigger**: 80% (24,800 tokens) +**Next action**: Trigger compaction at 24,800 tokens +``` + +## Evidence + +Context engineering patterns are observed in production agent runtimes where: +- Context budgets are explicit, not implicit +- Progressive disclosure reduces startup latency by 60-80% +- Manual cache invalidation prevents subtle staleness bugs +- Isolation patterns enable reliable multi-agent coordination diff --git a/.agents/skills/harness-creator/references/gotchas.md b/.agents/skills/harness-creator/references/gotchas.md new file mode 100644 index 000000000..c8cd51b78 --- /dev/null +++ b/.agents/skills/harness-creator/references/gotchas.md @@ -0,0 +1,209 @@ +# Gotchas — Harness Engineering Failure Modes + +Non-obvious principles that will cause bugs if you violate them. + +--- + +## 1. Memory Index Caps Fire Silently + +**Symptom**: Recent memories "disappear" without error. + +**Cause**: Index has hard caps (e.g., 200 lines / 25KB) enforced at read time. Long entries (multi-sentence summaries) hit byte cap while staying under line cap. + +**Fix**: Keep index entries to one-line hooks. Put detail in topic files. + +```markdown +✓ Good: "Use bun, not npm - user preference 2024-01-15" +✗ Bad: "The user prefers bun over npm because it's faster. This was discussed on 2024-01-15 when the user said 'use bun not npm' and I updated the package.json accordingly..." +``` + +--- + +## 2. Priority Ordering is Counterintuitive + +**Symptom**: Global rule silently overridden by local file. + +**Cause**: Local overrides beat project rules, which beat user rules, which beat org rules. If you inject at user level expecting it to dominate, a local override file in project root wins. + +**Fix**: Test with full instruction-file stack present: + +```bash +# Test priority ordering +cat ~/.claude/CLAUDE.md # User level +cat ./CLAUDE.md # Project level +cat ./CLAUDE.local.md # Local override (WINS) +``` + +--- + +## 3. Extraction Timing Creates Race Window + +**Symptom**: Background extractor writes memory, but user starts next turn before extraction completes. + +**Cause**: Extraction fires at end of response. User can send message before extraction finishes. + +**Fix**: Coalesce concurrent extraction requests. Advance cursor only after successful run. Failed extraction means those messages reconsidered next time. + +--- + +## 4. Derivable Content Doesn't Belong in Memory + +**Symptom**: Memory index fills with architecture details that stale quickly. + +**Cause**: Agent saves what's derivable from codebase (architecture, code patterns, version history). + +**Fix**: Exclude derivable content by design. Type taxonomy should forbid saving what's in the repo already. + +--- + +## 5. Concurrent Classification is Per-Call, Not Per-Tool + +**Symptom**: Tool marked "concurrent-safe" causes race conditions. + +**Cause**: Same tool can be safe for some inputs and unsafe for others. Don't assume tool's concurrency behavior is static. + +**Fix**: Classify each call at runtime: + +```typescript +// Don't do this: +toolRegistry.register('shell', { concurrentSafe: false }); + +// Do this: +function isCallConcurrentSafe(call: ToolCall): boolean { + if (call.args.command.startsWith('rm -rf')) return false; + if (call.args.command.startsWith('cat')) return true; + // ...runtime classification +} +``` + +--- + +## 6. Permission Evaluation Has Side Effects + +**Symptom**: Permission check changes behavior on subsequent calls. + +**Cause**: Permission evaluator tracks denials, transforms modes, updates state as side effect. Not a pure lookup function. + +**Fix**: Don't cache permission results across calls. Re-evaluate each call fresh. + +--- + +## 7. Most Async Work Skips "Pending" State + +**Symptom**: UI shows "pending" but work unit never enters that state. + +**Cause**: Work units register directly as "running" in practice. "Pending" exists in state machine but rarely used. + +**Fix**: Don't build UI that assumes every work unit starts pending. + +--- + +## 8. Fork Children Must Not Fork + +**Symptom**: Context cost explodes exponentially. + +**Cause**: Recursive forks multiply context: parent + child1 + child2 + grandchildren... + +**Fix**: Enforce single-level invariant. Keep fork tool in child's pool (for prompt cache sharing) but block at call time. + +--- + +## 9. Context Builders are Memoized but Manually Invalidated + +**Symptom**: Model sees stale data for entire session. + +**Cause**: Context builder cached at startup, but mutation doesn't clear cache. + +**Fix**: Every mutation point must explicitly clear its corresponding cache: + +```typescript +// Example: Cache invalidation at mutation point +async function editFile(path: string, content: string) { + await writeFile(path, content); + context.cache.invalidate(`file:${path}`); // MUST invalidate +} +``` + +--- + +## 10. Hook Trust is All-or-Nothing + +**Symptom**: Entire extension system disabled because one hook untrusted. + +**Cause**: If workspace untrusted, all hooks skip — not just suspicious ones. + +**Fix**: Design hooks with trust gate at dispatch point. Don't attempt per-hook trust evaluation. + +--- + +## 11. Eviction Requires Notification + +**Symptom**: Parent can never read work unit result. + +**Cause**: Work unit evicted before parent notified of completion. Race condition: parent tries to read result that's already GC'd. + +**Fix**: Two-phase eviction: +1. Clean disk output at terminal state (eager) +2. Clean in-memory record after parent notified (lazy) + +--- + +## 12. Skill Listing Budgets Are Tight + +**Symptom**: Skill description truncated, can't trigger properly. + +**Cause**: Skill descriptions concatenated and capped per entry (~150 chars). Front-loaded trigger language gets priority. + +**Fix**: Front-load distinctive trigger language: + +```markdown +✓ Good: "harness-patterns: Memory, permissions, context engineering, multi-agent" +✗ Bad: "A comprehensive skill for understanding and implementing various patterns related to AI agent harnesses and runtime systems..." +``` + +--- + +## 13. Default Tool Permission is "Allow" + +**Symptom**: Tool bypasses expected gate. + +**Cause**: Tools without custom permission logic delegate entirely to rule-based system. Default is "allow" unless configured otherwise. + +**Fix**: Override default for sensitive tools: + +```typescript +registry.register('shell', { + defaultPermission: 'ask', // NOT 'allow' + // ... +}); +``` + +--- + +## 14. Team Memory Requires Auto-Memory Enabled + +**Symptom**: Team-shared memory doesn't work even when configured. + +**Cause**: Team memory builds on same directory/index infrastructure as auto-memory. Disabling auto-memory (via env var or settings) also disables team memory. + +**Fix**: Ensure auto-memory enabled before enabling team memory. Check both feature gate and enablement check. + +--- + +## 15. Orphaned Topic Files Accumulate + +**Symptom**: Disk space fills with `.claude/memory/topics/` files. + +**Cause**: Two-step save (topic file then index). Crash between steps leaves orphaned topic file. + +**Fix**: Periodic sweep deletes topic files not referenced by index. Orphans don't corrupt index but consume disk space. + +--- + +## Related Reading + +- [Memory Persistence Pattern](memory-persistence-pattern.md) — Gotchas #1, #3, #4, #15 +- [Tool Registry Pattern](tool-registry-pattern.md) — Gotchas #5, #6, #13 +- [Multi-agent Pattern](multi-agent-pattern.md) — Gotchas #8, #11 +- [Context Engineering Pattern](context-engineering-pattern.md) — Gotchas #9 +- [Lifecycle Pattern](lifecycle-bootstrap-pattern.md) — Gotchas #10, #14 diff --git a/.agents/skills/harness-creator/references/lifecycle-bootstrap-pattern.md b/.agents/skills/harness-creator/references/lifecycle-bootstrap-pattern.md new file mode 100644 index 000000000..3fdb127c6 --- /dev/null +++ b/.agents/skills/harness-creator/references/lifecycle-bootstrap-pattern.md @@ -0,0 +1,264 @@ +# Lifecycle and Bootstrap Pattern + +## Problem + +Agent runtimes need extensibility without compromising safety: + +- **Hooks** — Extend behavior at lifecycle moments (pre/post tool execution, session start/end) +- **Background tasks** — Track long-running work without blocking the main agent +- **Bootstrap** — Structure initialization across multiple entry modes (CLI, server, SDK) + +But uncontrolled extensibility creates: +- Security holes from untrusted hooks +- Resource leaks from tasks that never complete +- Race conditions in initialization + +## Golden Rules + +### Hook Trust is All-or-Nothing + +If the workspace is untrusted, **all hooks skip** — not just suspicious ones. Session-scoped hooks are ephemeral and cleaned on session end. + +```typescript +// Example: Hook dispatch with trust gate +async function dispatchHook( + hookType: HookType, + context: HookContext +): Promise { + + // Trust gate: if workspace untrusted, skip ALL hooks + if (!context.trustBoundary.crossed) { + logger.warn('Untrusted workspace, skipping hooks'); + return []; + } + + // Session-scoped hooks ephemeral — cleanup on session end + const sessionHooks = context.hooks.getByScope('session'); + const projectHooks = context.hooks.getByScope('project'); + + return await Promise.all([ + ...sessionHooks.map(h => h.execute(context)), + ...projectHooks.map(h => h.execute(context)), + ]); +} +``` + +### Long-Running Work: Typed State Machines with Two-Phase Eviction + +Each work unit gets: +1. **Typed, prefixed ID** (e.g., `extractor-001`, `benchmark-002`) +2. **Strict lifecycle** (running → completed | failed | killed) +3. **Disk-backed output** (not just in-memory) + +Eviction is two-phase: +1. **Disk output** cleaned eagerly at terminal state +2. **In-memory records** cleaned lazily after parent notified + +### Bootstrap: Dependency-Ordered, Memoized Stages + +Multiple entry modes (CLI, server, SDK) share the same bootstrap path: + +``` +Stage 1: Create minimal context (no trust required) + ↓ +Stage 2: Load tools (read-only safe) + ↓ +Stage 3: Trust boundary crossed (user grants consent) + ↓ +Stage 4: Load security-sensitive subsystems (telemetry, secret env vars) +``` + +**Critical inflection**: Security-sensitive subsystems must not activate before trust is established. + +## When To Use + +- You need to extend agent behavior without modifying core code +- You need to track long-running background work +- You need structured initialization across multiple entry modes +- You need hooks at lifecycle moments (pre/post tool, session start/end) + +## Tradeoffs + +| Decision | Benefit | Cost | +|---|---|---| +| All-or-nothing hook trust | Simple security boundary | One untrusted hook disables entire extension system | +| Disk-backed task output | Memory constant regardless of concurrent work | I/O latency proportional to work units | +| Dependency-ordered bootstrap | Multiple entry modes share path | Initial startup sequential (can't parallelize stages) | +| Memoized stages | Re-init is fast | Must carefully invalidate memoization on config change | + +## Implementation Patterns + +### Hook Lifecycle + +Six hook types dispatched at defined moments: + +```typescript +interface HookRegistry { + // Session lifecycle + onSessionStart: (context: SessionContext) => Promise; + onSessionEnd: (context: SessionContext) => Promise; + + // Tool execution + preToolExecute: (context: ToolContext) => Promise; + postToolExecute: (context: ToolResult) => Promise; + + // Prompt submission + prePromptSubmit: (context: PromptContext) => Promise; + postPromptSubmit: (context: ResponseContext) => Promise; +} + +// Usage: Register hooks via config +// /update-config hooks.preToolExecute = "scripts/audit-tool-call.js" +``` + +### Long-Running Task Tracking + +```typescript +interface TaskRegistry { + // Typed prefixed IDs + registerWork( + type: 'extraction' | 'benchmark' | 'indexing', + outputType: 'json' | 'text' | 'file' + ): string; // Returns typed ID: `extraction-001` + + // Strict state machine + updateState( + taskId: string, + state: 'running' | 'completed' | 'failed' | 'killed', + output?: any + ): void; + + // Two-phase eviction + evictTask(taskId: string): void; + // 1. Clean disk output (eager, at terminal state) + // 2. Clean in-memory record (lazy, after parent notified) +} +``` + +### Bootstrap Sequence + +```typescript +// Example: Dependency-ordered initialization +class AgentBootstrap { + private stages = new Map(); + private memoizedCallers = new Map(); + + async bootstrap(entryMode: 'cli' | 'server' | 'sdk'): Promise { + + // Stage 1: Minimal context (no trust required) + await this.runStage('minimal-context', async () => { + return { + cwd: process.cwd(), + entryMode, + trustBoundary: { crossed: false }, + }; + }); + + // Stage 2: Load tools (read-only safe) + await this.runStage('load-tools', async (context) => { + context.tools = await this.loadSafeTools(); + return context; + }); + + // Stage 3: Trust boundary (user grants consent) + await this.runStage('trust-boundary', async (context) => { + const consent = await this.requestConsent(); + context.trustBoundary = { crossed: consent }; + return context; + }); + + // Stage 4: Security-sensitive subsystems (requires trust) + if (context.trustBoundary.crossed) { + await this.runStage('load-sensitive', async (context) => { + context.telemetry = await this.loadTelemetry(); + context.secretEnvVars = await this.loadSecrets(); + return context; + }); + } + + return context; + } + + private async runStage( + name: string, + fn: (context: AgentContext) => Promise + ): Promise { + // Memoized: skip if already run + if (this.stages.has(name) && this.stages.get(name).complete) { + return; + } + + // Run stage + const stage = { name, complete: false, running: true }; + this.stages.set(name, stage); + + try { + await fn(this.context); + stage.complete = true; + } finally { + stage.running = false; + } + } +} +``` + +## Gotchas + +1. **Hook trust is all-or-nothing** — One untrusted hook disables entire extension system +2. **Most async work skips "pending" state** — Work units register directly as "running" +3. **Eviction requires notification** — Terminal work unit only GC-eligible after parent notified +4. **Fast-path dispatch** — Memoized callers must handle concurrent calls without re-running stages +5. **Hook types must be disjoint** — Don't create overlapping hook scopes + +## Related Patterns + +- [Tool Registry](tool-registry-pattern.md) — How tools are registered at bootstrap +- [Memory Persistence](memory-persistence-pattern.md) — How memory is loaded at init + +## Template: Bootstrap Checklist + +Before declaring bootstrap complete: + +```markdown +## Bootstrap Verification + +### Stage 1: Minimal Context +- [ ] Working directory confirmed +- [ ] Entry mode determined (cli / server / sdk) +- [ ] Trust boundary NOT crossed (no secrets loaded) + +### Stage 2: Tools Loaded +- [ ] Read-only tools registered (read, search, glob) +- [ ] Write tools NOT yet registered (edit, shell) +- [ ] Tool permissions set to default (ask / deny) + +### Stage 3: Trust Boundary +- [ ] User consent requested (interactive or config flag) +- [ ] Consent recorded in session state +- [ ] Security audit logged + +### Stage 4: Sensitive Subsystems +- [ ] Telemetry initialized (if consent given) +- [ ] Secret env vars loaded (if consent given) +- [ ] Write tools registered (edit, shell, exec) +- [ ] Hook system enabled (if workspace trusted) + +### Stage 5: Background Tasks +- [ ] Task registry initialized +- [ ] Cleanup handlers registered +- [ ] Drain-on-shutdown configured + +## If Any Stage Fails + +- Bootstrap halts immediately +- Session remains in safe mode (read-only) +- Error logged with stage name and failure reason +``` + +## Evidence + +Lifecycle and bootstrap patterns are observed in production runtimes where: +- Hook dispatch is all-or-nothing based on workspace trust +- Long-running tasks use typed prefixed IDs and disk-backed output +- Bootstrap is dependency-ordered with memoized stages +- Trust boundary is explicit inflection point for security-sensitive subsystems diff --git a/.agents/skills/harness-creator/references/memory-persistence-pattern.md b/.agents/skills/harness-creator/references/memory-persistence-pattern.md new file mode 100644 index 000000000..089e0ac7e --- /dev/null +++ b/.agents/skills/harness-creator/references/memory-persistence-pattern.md @@ -0,0 +1,110 @@ +# Memory and Persistence Pattern + +## Problem + +Without persistent memory, an agent loses all user preferences, project context, and behavioral feedback the moment a session ends. Users must repeat corrections every session ("use bun, not npm"), and the agent cannot accumulate the working knowledge that makes it genuinely useful over time. + +## Golden Rules + +### Separate layers by scope and durability + +- **Instruction memory** (human-curated, version-controlled): AGENTS.md, CLAUDE.md, project conventions +- **Auto-memory** (agent-written, persistent): Progress logs, session handoffs, discovered patterns +- **Session extraction** (background-derived): Automatic transcript analysis at session end + +### Two-step save invariant + +Every memory write is a two-step operation: +1. Write the full content to a dedicated topic file +2. Append a one-line pointer to the index + +If the process crashes between steps, the worst outcome is an orphaned topic file — the index remains consistent. + +### Local overrides win — always + +When the same topic is addressed at multiple scopes, the most-local instruction takes priority: + +``` +Organization-wide → User-level → Project-level → Local override + ↓ ↓ ↓ ↓ + sets floor narrows it narrows further final say +``` + +### The index is bounded always-on context; topic files are on-demand detail + +- **Index**: Hard-capped at ~200 lines / 25KB, one line per entry +- **Topic files**: Unlimited detail, loaded on demand + +## When To Use + +- Your agent persists across sessions and must recall user preferences or project context +- Multiple scopes of instruction coexist and need clear priority ordering +- The agent should learn from sessions without manual curation +- You need background extraction that doesn't block the user + +## Tradeoffs + +| Decision | Benefit | Cost | +|---|---|---| +| Layered memory | Each scope can be shared, audited, overridden independently | More files to discover at startup | +| Local-wins priority | Users can override without touching shared files | Global rule can be silently overridden | +| Bounded index with on-demand topics | Constant context cost regardless of memory volume | Agent must perform extra retrieval step | +| Background extraction | No latency added to user responses | Race window between extraction and next turn | + +## Implementation Patterns + +1. **Define memory directory** idempotently at startup (e.g., `.claude/memory/`) +2. **Create index file** with hard caps enforced at read time +3. **Implement two-step save**: topic file first, then index update +4. **Fire background extraction** only after final response with no pending tool calls +5. **Enforce mutual exclusion**: if main agent wrote to memory, skip extraction that turn +6. **Build review mechanism** for cross-layer promotion proposals + +## Gotchas + +1. **Index truncation is silent until it fires** — keep entries short +2. **Priority ordering is counterintuitive** — local beats project beats user beats org +3. **Extraction timing creates a race window** — user can start next turn before extraction completes +4. **Derivable content doesn't belong in memory** — architecture and code patterns are re-derivable from the codebase +5. **Orphaned topic files accumulate** — periodic cleanup recommended + +## Related Patterns + +- [Context Engineering](context-engineering-pattern.md) — How to manage context budget across layers +- [Lifecycle & Bootstrap](lifecycle-bootstrap-pattern.md) — How initialization loads memory + +## Template: Progress Log Structure + +```markdown +# Session Progress Log + +## Current State (Last Updated: YYYY-MM-DD HH:MM) + +**Active Feature:** feat-003 - Q&A with Citations +**Status:** In Progress (60% complete) + +### What's Done +- [x] Document chunking pipeline +- [x] Index data structure +- [ ] Q&A handler (in progress) + +### What's In Progress +- Implementing Q&A IPC handler +- Need to decide: streaming vs batch response + +### Blockers +- Waiting on decision: citation format (footnotes vs inline) + +### Next Session Should +1. Complete Q&A handler +2. Add citation formatting +3. Test end-to-end flow +``` + +## Evidence + +This pattern is grounded in production agent runtimes including Claude Code's memory system, which implements: +- Four-level instruction hierarchy (org/user/project/local) +- Four-type auto-memory taxonomy (user/feedback/project/reference) +- Background session extraction with mutual exclusion +- Team-shared memory as an extension layer diff --git a/.agents/skills/harness-creator/references/multi-agent-pattern.md b/.agents/skills/harness-creator/references/multi-agent-pattern.md new file mode 100644 index 000000000..36a0aa3f8 --- /dev/null +++ b/.agents/skills/harness-creator/references/multi-agent-pattern.md @@ -0,0 +1,193 @@ +# Multi-Agent Coordination Pattern + +## Problem + +Single agents hit limits: +- **Context limits** — Can't hold full research + implementation in one session +- **Specialization** — Need separate researchers, implementers, reviewers +- **Parallelism** — Want to explore multiple approaches simultaneously + +But multi-agent systems introduce chaos: +- Workers duplicate each other's research +- Coordinators delegate understanding instead of synthesizing +- Context inheritance explodes exponentially + +## Golden Rules + +### The Coordinator Must Synthesize, Not Delegate Understanding + +**Anti-pattern:** +> "Based on your findings, fix the authentication system." + +**Pattern:** +> "Research identified 3 auth flows: login, logout, token refresh. Implement ONLY the token refresh handler using the JWT strategy documented in [research output]. Return: implementation diff + test results." + +The coordinator (orchestrator) adds value by digesting worker results into precise specs before dispatching implementation. + +### Three Delegation Patterns + +| Pattern | Context Sharing | Best For | Constraints | +|---------|----------------|----------|-------------| +| **Coordinator** | None — workers start fresh | Complex multi-phase tasks (research → synthesize → implement → verify) | Slowest but safest | +| **Fork** | Full — child inherits parent history | Quick parallel splits sharing loaded context | **Single-level only** — recursive forks multiply context cost | +| **Swarm** | Peer-to-peer via shared task list | Long-running independent workstreams | **Flat roster** — teammates can't spawn other teammates | + +### Results Arrive Asynchronously; Fire-and-Forget Registration Returns ID Immediately + +```typescript +// Example: Spawn worker, get ID back immediately +const taskId = await coordinator.spawn({ + type: 'research', + prompt: 'Analyze auth flows...', + toolFilter: ['read', 'search'], // Restrict tools +}); + +// Parent can continue working while worker runs +// Results arrive via callback or polling +``` + +## When To Use + +- Task too large for single agent session +- Need parallel exploration (e.g., prototype multiple approaches) +- Want persistent specialized teammates (researcher, implementer, reviewer) +- Complex multi-phase workflows + +## Tradeoffs + +| Pattern | Speed | Safety | Context Cost | +|---------|-------|--------|--------------| +| **Coordinator** | Slowest | Safest | Lowest (zero inheritance) | +| **Fork** | Fastest | Medium | Highest (full inheritance) | +| **Swarm** | Medium | Medium | Medium (shared state only) | + +## Implementation Patterns + +### Coordinator Pattern (Recommended for Complex Tasks) + +Phased workflow: + +``` +Phase 1: Research + ↓ (synthesize findings) +Phase 2: Plan + ↓ (precise specs) +Phase 3: Implement + ↓ (verify) +Phase 4: Review +``` + +```typescript +// Example: Coordinator workflow +const research = await coordinator.spawn({ + role: 'researcher', + prompt: `Analyze existing authentication in ${authDir}. + Find: login flow, logout flow, token handling. + Return: structured findings only. NO implementation suggestions.`, + toolFilter: ['read', 'search', 'glob'], // Can't write +}); + +await coordinator.synthesize(research.results); + +const implement = await coordinator.spawn({ + role: 'implementer', + prompt: `Implement token refresh handler using the JWT strategy + from [Phase 2 findings]. + Constraints: Use existing AuthService patterns, add tests.`, + toolFilter: ['read', 'search', 'edit', 'test'], // Can write +}); +``` + +### Fork Pattern (Single-Level Only) + +```typescript +// Parent spawns children for parallel work +const forks = await Promise.all([ + coordinator.fork({ + prompt: 'Implement login handler', + inheritContext: true, // Full parent history + }), + coordinator.fork({ + prompt: 'Implement logout handler', + inheritContext: true, + }), +]); + +// CRITICAL: Children must not fork recursively +// If allowed, context cost multiplies: parent + child1 + child2 + ... +``` + +### Swarm Pattern (Flat Roster) + +```typescript +// Swarm: persistent team with shared task list +const swarm = new Swarm([ + { id: 'researcher', specialty: 'research' }, + { id: 'implementer', specialty: 'implementation' }, + { id: 'reviewer', specialty: 'verification' }, +]); + +// Agents pick tasks from shared queue +// Results posted back to shared state +await swarm.dispatch({ + taskId: 'feat-001', + pickedBy: 'implementer', +}); +``` + +## Gotchas + +1. **Fork children must not fork** — Recursive guard preserves single-level invariant. Keep fork tool in child's pool (for prompt cache sharing) but block at call time. +2. **Coordinator workers start with zero context** — Only explicit prompt is passed. Don't assume child sees parent's accumulated research. +3. **Swarm teammates cannot spawn other teammates** — Roster is flat to prevent uncontrolled growth. +4. **Write self-contained prompts** — "Based on your findings" is an anti-pattern. Coordinator must digest first. +5. **Filter each worker's tool set** — Researcher doesn't need write; implementer doesn't need broad search. + +## Related Patterns + +- [Context Engineering](context-engineering-pattern.md) — Isolation patterns for delegation +- [Lifecycle & Bootstrap](lifecycle-bootstrap-pattern.md) — How agents are spawned at init + +## Template: Worker Prompt Structure + +```markdown +# Self-Contained Worker Prompt + +## Context (Copied from Coordinator Synthesis) + +**Task**: Implement token refresh handler +**Background**: Research identified JWT-based auth with 24h access tokens. +**Decision**: Use refresh token rotation (new refresh token on each refresh). + +## Your Role + +You are an **implementer**. Your job is to write production code following the specs above. + +## Constraints + +- Use existing patterns from `${authServicePath}` +- Add tests for success and failure cases +- Do NOT modify login/logout handlers (separate task) + +## Your Tools + +- read, search, edit, test +- Shell: npm test, npm run check only + +## Deliverable + +Return: +1. Implementation diff (files changed) +2. Test results (pass/fail) +3. Any blockers or clarifications needed + +**Do NOT return**: Research findings, architectural debates, alternative designs. +``` + +## Evidence + +Multi-agent coordination patterns are observed in production systems where: +- Coordinator workers start with zero context inheritance +- Fork is restricted to single-level to control context explosion +- Swarm agents communicate through shared task lists, not direct prompts +- Results arrive asynchronously with fire-and-forget registration diff --git a/.agents/skills/harness-creator/references/skill-runtime-pattern.md b/.agents/skills/harness-creator/references/skill-runtime-pattern.md new file mode 100644 index 000000000..963d98ec6 --- /dev/null +++ b/.agents/skills/harness-creator/references/skill-runtime-pattern.md @@ -0,0 +1,43 @@ +# Skill Runtime Pattern + +Use this pattern when you want to package reusable agent behavior as a skill instead of repeating long instructions in every repository. + +## What Belongs in a Skill + +- Reusable workflows that apply across projects. +- Domain-specific decision procedures. +- Templates, checklists, and reference material the agent should load on demand. +- Small helper scripts when they are stable and safe to run. + +## What Does Not Belong in a Skill + +- Project-specific architecture facts that should live in the target repository. +- Secrets, tokens, private URLs, or user-specific credentials. +- Large manuals that the agent must always read before acting. +- Commands with destructive side effects unless they are clearly documented and require explicit user approval. + +## Runtime Shape + +A production skill should use progressive disclosure: + +1. `SKILL.md` frontmatter explains when the skill should trigger. +2. The body gives the shortest reliable workflow. +3. `references/` contains deeper material loaded only when relevant. +4. `templates/` contains copyable artifacts. +5. `evals/` captures representative quality checks. + +## Design Rules + +- Keep the entry file concise enough to scan quickly. +- Prefer concrete checklists over abstract advice. +- Link every referenced bundled file and verify it exists. +- Make installation instructions explicit about the repository, skill name, and target agent. +- Treat scripts as optional helpers, not hidden behavior. + +## Validation Checklist + +- [ ] `SKILL.md` exists and has valid frontmatter. +- [ ] Every referenced file exists inside the skill directory. +- [ ] Templates are safe to copy into a target repository. +- [ ] Installation command has been tested with `skills add --list` or equivalent. +- [ ] The skill does not depend on private local paths. diff --git a/.agents/skills/harness-creator/references/tool-registry-pattern.md b/.agents/skills/harness-creator/references/tool-registry-pattern.md new file mode 100644 index 000000000..1a38db9cd --- /dev/null +++ b/.agents/skills/harness-creator/references/tool-registry-pattern.md @@ -0,0 +1,200 @@ +# Tool Registry and Safety Pattern + +## Problem + +Agents need tools (shell, file edit, search, etc.) to be productive. But unbounded tool access creates risks: + +- Destructive operations (rm -rf, DROP TABLE, etc.) +- Race conditions from concurrent tool calls +- Silent policy violations from misconfigured permissions + +The solution is a **fail-closed registry** with explicit concurrency classification and a multi-source permission pipeline. + +## Golden Rules + +### Default to Fail-Closed + +Tools are **non-concurrent** and **non-read-only** unless explicitly marked safe. This prevents: +- Accidental parallel execution of state-mutating operations +- Silent data corruption from concurrent writes + +### Concurrency is Per-Call, Not Per-Tool + +The same tool can be safe for some inputs and unsafe for others: + +``` +✓ Safe (can run in parallel): + - cat file1.txt + - grep "pattern" src/ + - ls -la + +✗ Unsafe (must run serially): + - rm -rf build/ + - npm install (network, filesystem mutation) + - sed -i 's/old/new/g' *.ts +``` + +The runtime partitions a batch of tool calls into consecutive groups: safe calls run in parallel; any unsafe call starts a serial segment. + +### Permission Pipeline has Side Effects + +The permission evaluator is **stateful** — it: +- Tracks denials (for audit and rate limiting) +- Transforms modes (e.g., auto → ask after denial) +- Updates session state as a side effect + +**Strict priority order:** +``` +Policy (org-wide) → User settings → Project rules → Local overrides → Session grants +``` + +## When To Use + +- Your agent runtime needs tool registration +- You need concurrency control for parallel tool calls +- You need permission gating (auto-approve, ask-first, deny) +- You need to track tool usage for audit + +## Tradeoffs + +| Decision | Benefit | Cost | +|---|---|---| +| Fail-closed defaults | New tools are safe out of the box | Developers must actively opt into concurrency | +| Per-call classification | Fine-grained control over parallelism | Requires analyzing each call, not just tool registration | +| Multi-source permission layering | Flexible policy composition | Hard to debug when rules conflict | +| Stateful evaluator | Can adapt behavior based on history | Not a pure function — harder to test | + +## Implementation Patterns + +### Tool Registration + +```typescript +// Example: Tool registry entry +interface ToolDefinition { + name: string; + description: string; + handler: (args: any) => Promise; + + // Safety classification + isReadOnly: boolean; // Default: false + isConcurrentSafe: boolean; // Default: false + + // Optional custom permission logic + permissionCheck?: (args: any, context: ToolContext) => PermissionResult; +} + +// Register tools +registry.register('read_file', { + name: 'read_file', + description: 'Read contents of a file', + handler: readFile, + isReadOnly: true, + isConcurrentSafe: true, // Safe to read multiple files in parallel +}); + +registry.register('write_file', { + name: 'write_file', + description: 'Write or overwrite a file', + handler: writeFile, + isReadOnly: false, + isConcurrentSafe: false, // Must run serially to prevent race conditions +}); +``` + +### Permission Pipeline + +```typescript +// Permission evaluation order +async function evaluatePermission( + toolCall: ToolCall, + context: PermissionContext +): Promise { + + // 1. Policy rules (highest priority, org-wide) + const policyResult = await policyEngine.check(toolCall, context); + if (policyResult !== 'defer') return policyResult; + + // 2. User settings + const userResult = await userSettings.check(toolCall, context); + if (userResult !== 'defer') return userResult; + + // 3. Project rules + const projectResult = await projectRules.check(toolCall, context); + if (projectResult !== 'defer') return projectResult; + + // 4. Local overrides + const localResult = await localOverrides.check(toolCall, context); + if (localResult !== 'defer') return localResult; + + // 5. Session grants (lowest priority) + return sessionGrants.check(toolCall, context); +} +``` + +### Bypass-Immune Rules + +Certain paths or operations should never be auto-approved: + +```yaml +# Protected paths (never auto-approve) +protected_paths: + - /etc/** + - /usr/** + - node_modules/** + - .git/** + +# Protected commands (always ask) +protected_commands: + - "rm -rf*" + - "DROP TABLE*" + - "DELETE FROM*" + - "mkfs*" +``` + +## Gotchas + +1. **Most async work skips "pending" state** — work units register directly as "running" +2. **Permission evaluation has side effects** — don't cache results across calls +3. **Concurrency classification requires analyzing inputs**, not just tool name +4. **The default permission for tools is "allow"** — tools without custom logic delegate to rule-based system +5. **Eviction requires notification** — terminal work units only GC-eligible after parent notified + +## Related Patterns + +- [Lifecycle & Bootstrap](lifecycle-bootstrap-pattern.md) — How tools are registered at init +- Hook Lifecycle (`hook-lifecycle-pattern.md`) — Pre/post tool execution hooks + +## Template: Tool Safety Checklist + +Before enabling a new tool: + +```markdown +## Tool Safety Review + +**Tool name**: [e.g., execute_shell] + +### Classification +- [ ] Determined if read-only (true / false / depends on args) +- [ ] Determined if concurrent-safe (true / false / depends on args) +- [ ] Documented unsafe input patterns + +### Permission Requirements +- [ ] Default mode set to "ask" or "deny" +- [ ] Bypass-immune paths/commands defined +- [ ] Custom permission logic implemented (if needed) +- [ ] Audit logging enabled + +### Testing +- [ ] Tested with safe inputs (should auto-approve) +- [ ] Tested with unsafe inputs (should ask/deny) +- [ ] Tested concurrent execution (should serialize if unsafe) +- [ ] Tested error handling (failures logged, state consistent) +``` + +## Evidence + +Tool registry and safety patterns are observed in production agent runtimes including: +- Claude Code's tool registry with explicit concurrency flags +- Multi-source permission evaluation (settings → project → session) +- Protected path/command lists that bypass auto-approve modes +- Per-call concurrency classification that partitions tool batches diff --git a/.agents/skills/harness-creator/scripts/create-harness.mjs b/.agents/skills/harness-creator/scripts/create-harness.mjs new file mode 100644 index 000000000..1a33a7f64 --- /dev/null +++ b/.agents/skills/harness-creator/scripts/create-harness.mjs @@ -0,0 +1,75 @@ +#!/usr/bin/env node +import { chmod, mkdir } from 'node:fs/promises'; +import path from 'node:path'; +import { + copyTemplate, + detectPackageManager, + detectProject, + exists, + initScriptFromCommands, + parseArgs, + verificationCommands, + writeText +} from './lib/harness-utils.mjs'; + +const args = parseArgs(process.argv.slice(2)); + +if (args.help) { + console.log(`Usage: node scripts/create-harness.mjs [--target DIR] [--agent-file AGENTS.md|CLAUDE.md] [--package-manager npm|pnpm|yarn|bun] [--force] + +Creates a minimal production harness: + AGENTS.md or CLAUDE.md + feature_list.json + progress.md + session-handoff.md + init.sh + +Existing files are skipped unless --force is set.`); + process.exit(0); +} + +const target = path.resolve(args.target || args._[0] || process.cwd()); +const agentFile = args.agentFile || 'AGENTS.md'; +const force = Boolean(args.force); +const project = await detectProject(target); +project.packageManager = detectPackageManager(target, args.packageManager); +const commands = args.commands + ? String(args.commands).split(',').map((command) => command.trim()).filter(Boolean) + : verificationCommands(project, args.packageManager); + +await mkdir(target, { recursive: true }); + +const replacements = { + AGENT_FILE_NAME: agentFile, + PROJECT_PURPOSE: project.stack === 'generic' + ? 'Project harness for reliable agent-assisted development.' + : `Project harness for reliable agent-assisted development in a ${project.stack} codebase.`, + VERIFICATION_COMMANDS: commands.map((command) => `- \`${command}\``).join('\n'), + PRIMARY_VERIFICATION_COMMAND: './init.sh' +}; + +const results = []; +results.push(await copyTemplate('agents.md', path.join(target, agentFile), replacements, { force })); +results.push(await copyTemplate('feature-list.json', path.join(target, 'feature_list.json'), {}, { force })); +results.push(await copyTemplate('progress.md', path.join(target, 'progress.md'), {}, { force })); +results.push(await copyTemplate('session-handoff.md', path.join(target, 'session-handoff.md'), {}, { force })); + +const initPath = path.join(target, 'init.sh'); +if (force || !await exists(initPath)) { + await writeText(initPath, initScriptFromCommands(commands)); + await chmod(initPath, 0o755); + results.push({ path: initPath, status: 'written' }); +} else { + results.push({ path: initPath, status: 'skipped', reason: 'exists' }); +} + +console.log(`Created harness for ${target}`); +console.log(`Detected stack: ${project.stack}`); +console.log(`Verification commands:`); +for (const command of commands) { + console.log(` - ${command}`); +} +console.log(''); +for (const result of results) { + console.log(`${result.status.toUpperCase()} ${path.relative(target, result.path)}${result.reason ? ` (${result.reason})` : ''}`); +} diff --git a/.agents/skills/harness-creator/scripts/render-assessment-html.mjs b/.agents/skills/harness-creator/scripts/render-assessment-html.mjs new file mode 100644 index 000000000..4d8d85fdb --- /dev/null +++ b/.agents/skills/harness-creator/scripts/render-assessment-html.mjs @@ -0,0 +1,26 @@ +#!/usr/bin/env node +import path from 'node:path'; +import { + htmlReport, + loadHarnessFiles, + parseArgs, + scoreHarness, + writeText +} from './lib/harness-utils.mjs'; + +const args = parseArgs(process.argv.slice(2)); + +if (args.help) { + console.log(`Usage: node scripts/render-assessment-html.mjs [--target DIR] [--output FILE] + +Renders the five-subsystem harness assessment as a standalone HTML file.`); + process.exit(0); +} + +const target = path.resolve(args.target || args._[0] || process.cwd()); +const output = path.resolve(args.output || path.join(target, 'harness-assessment.html')); +const result = scoreHarness(await loadHarnessFiles(target)); + +await writeText(output, htmlReport(result, `Harness Assessment: ${path.basename(target)}`)); +console.log(`HTML report written to ${output}`); +console.log(`Overall: ${result.overall}/100`); diff --git a/.agents/skills/harness-creator/scripts/run-benchmark.mjs b/.agents/skills/harness-creator/scripts/run-benchmark.mjs new file mode 100644 index 000000000..2e6ab0f6c --- /dev/null +++ b/.agents/skills/harness-creator/scripts/run-benchmark.mjs @@ -0,0 +1,121 @@ +#!/usr/bin/env node +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { + formatScoreReport, + htmlReport, + loadHarnessFiles, + parseArgs, + readJson, + scoreHarness, + writeText +} from './lib/harness-utils.mjs'; + +const args = parseArgs(process.argv.slice(2)); +const scriptDir = path.dirname(fileURLToPath(import.meta.url)); +const skillRoot = path.resolve(scriptDir, '..'); + +if (args.help) { + console.log(`Usage: node scripts/run-benchmark.mjs [--target DIR] [--output FILE] [--html FILE] + +Runs a lightweight harness benchmark: + 1. Scores the current target harness. + 2. Checks eval coverage in evals/evals.json. + 3. Produces a JSON report and optional HTML report. + +This is a structural benchmark, not an LLM judge. Use it before/after real agent sessions.`); + process.exit(0); +} + +const target = path.resolve(args.target || args._[0] || process.cwd()); +const output = path.resolve(args.output || path.join(target, 'harness-benchmark.json')); +const evalPath = path.resolve(args.evals || path.join(skillRoot, 'evals', 'evals.json')); + +const harnessResult = scoreHarness(await loadHarnessFiles(target)); +const evals = await readJson(evalPath); +const evalResult = scoreEvals(evals); +const report = { + generatedAt: new Date().toISOString(), + target, + harness: harnessResult, + evals: evalResult, + recommendation: recommend(harnessResult, evalResult) +}; + +await writeText(output, `${JSON.stringify(report, null, 2)}\n`); +console.log(`Benchmark report written to ${output}`); +console.log(''); +console.log(formatScoreReport(harnessResult, target)); +console.log(`Eval coverage: ${evalResult.score}/100 (${evalResult.passed}/${evalResult.total})`); +console.log(`Recommendation: ${report.recommendation}`); + +if (args.html) { + const htmlPath = path.resolve(args.html); + await writeText(htmlPath, renderBenchmarkHtml(report)); + console.log(`HTML benchmark report written to ${htmlPath}`); +} + +if (harnessResult.overall < Number(args.minScore || 70) || evalResult.score < Number(args.minEvalScore || 80)) { + process.exitCode = 1; +} + +function scoreEvals(evalsJson) { + const cases = Array.isArray(evalsJson.evals) ? evalsJson.evals : []; + const checks = []; + checks.push({ pass: cases.length >= 10, message: 'At least 10 eval cases' }); + checks.push({ pass: cases.some((item) => /minimal|creation/i.test(item.name)), message: 'Covers minimal harness creation' }); + checks.push({ pass: cases.some((item) => /session|continuity/i.test(item.name)), message: 'Covers session continuity' }); + checks.push({ pass: cases.some((item) => /assessment|score/i.test(item.name)), message: 'Covers harness assessment' }); + checks.push({ pass: cases.some((item) => /verification/i.test(item.name)), message: 'Covers verification workflow' }); + checks.push({ pass: cases.some((item) => /memory/i.test(item.name)), message: 'Covers memory taxonomy' }); + checks.push({ pass: cases.some((item) => /tool|permission|safety/i.test(item.name)), message: 'Covers tool safety' }); + checks.push({ pass: cases.some((item) => /multi-agent|delegation|coordination/i.test(item.name)), message: 'Covers multi-agent coordination' }); + checks.push({ pass: cases.every((item) => item.prompt && item.expected_output && Array.isArray(item.expectations)), message: 'Each eval has prompt, expected output, expectations' }); + checks.push({ pass: cases.every((item) => item.expectations?.length >= 3), message: 'Each eval has at least three expectation checks' }); + + const passed = checks.filter((check) => check.pass).length; + return { + score: Math.round((passed / checks.length) * 100), + passed, + total: checks.length, + cases: cases.length, + checks + }; +} + +function recommend(harnessResult, evalResult) { + if (harnessResult.overall >= 85 && evalResult.score >= 90) { + return 'Ready for realistic before/after agent-session benchmarking.'; + } + if (harnessResult.overall < 70) { + return `Improve the ${harnessResult.bottleneck} subsystem before benchmarking agent behavior.`; + } + if (evalResult.score < 80) { + return 'Expand eval coverage before treating benchmark results as representative.'; + } + return 'Usable, with some gaps worth tightening after first real sessions.'; +} + +function renderBenchmarkHtml(report) { + const evalHtml = htmlReport(report.harness, `Harness Benchmark: ${path.basename(report.target)}`) + .replace('', `
+

Eval Coverage ${report.evals.score}/100

+

${report.evals.passed}/${report.evals.total} benchmark checks passed across ${report.evals.cases} eval cases.

+
    ${report.evals.checks.map((check) => `
  • ${check.pass ? 'PASS' : 'FAIL'} ${escapeHtml(check.message)}
    • `).join('')}
+
+
+

Recommendation

+

${escapeHtml(report.recommendation)}

+
+ `); + return evalHtml; +} + +function escapeHtml(value) { + return String(value) + .replaceAll('&', '&') + .replaceAll('<', '<') + .replaceAll('>', '>') + .replaceAll('"', '"') + .replaceAll("'", '''); +} diff --git a/.agents/skills/harness-creator/scripts/validate-harness.mjs b/.agents/skills/harness-creator/scripts/validate-harness.mjs new file mode 100644 index 000000000..3ae16d30f --- /dev/null +++ b/.agents/skills/harness-creator/scripts/validate-harness.mjs @@ -0,0 +1,43 @@ +#!/usr/bin/env node +import path from 'node:path'; +import { + formatScoreReport, + htmlReport, + loadHarnessFiles, + parseArgs, + scoreHarness, + writeText +} from './lib/harness-utils.mjs'; + +const args = parseArgs(process.argv.slice(2)); + +if (args.help) { + console.log(`Usage: node scripts/validate-harness.mjs [--target DIR] [--json] [--html FILE] + +Scores a project harness across five subsystems: + instructions, state, verification, scope, lifecycle + +Exit code is 0 when the harness scores at least --min-score (default 70).`); + process.exit(0); +} + +const target = path.resolve(args.target || args._[0] || process.cwd()); +const minScore = Number(args.minScore || 70); +const files = await loadHarnessFiles(target); +const result = scoreHarness(files); + +if (args.html) { + const htmlPath = path.resolve(args.html); + await writeText(htmlPath, htmlReport(result, `Harness Assessment: ${path.basename(target)}`)); + console.log(`HTML report written to ${htmlPath}`); +} + +if (args.json) { + console.log(JSON.stringify(result, null, 2)); +} else { + console.log(formatScoreReport(result, target)); +} + +if (result.overall < minScore) { + process.exitCode = 1; +} diff --git a/.agents/skills/harness-creator/templates/agents.md b/.agents/skills/harness-creator/templates/agents.md new file mode 100644 index 000000000..0b41e940b --- /dev/null +++ b/.agents/skills/harness-creator/templates/agents.md @@ -0,0 +1,68 @@ +# {{AGENT_FILE_NAME}} + +{{PROJECT_PURPOSE}} + +## Startup Workflow + +Before writing code: + +1. **Confirm working directory** with `pwd` +2. **Read this file** completely +3. **Read project docs if present** (`docs/ARCHITECTURE.md`, `docs/PRODUCT.md`, README, or equivalent) +4. **Run `./init.sh`** to verify environment is healthy +5. **Read `feature_list.json`** to see current feature state +6. **Review recent commits** with `git log --oneline -5` + +If baseline verification is failing, repair that first before adding new scope. + +## Working Rules + +- **One feature at a time**: Pick exactly one unfinished feature from `feature_list.json` +- **Verification required**: Don't claim done without running verification commands +- **Update artifacts**: Before ending session, update `progress.md` and `feature_list.json` +- **Stay in scope**: Don't modify files unrelated to the current feature +- **Leave clean state**: Next session must be able to run `./init.sh` immediately + +## Required Artifacts + +- `feature_list.json` — Feature state tracker (source of truth) +- `progress.md` — Session continuity log +- `init.sh` — Standard startup and verification path +- `session-handoff.md` — Optional, for larger sessions + +## Definition of Done + +A feature is done only when ALL of the following are true: + +- [ ] Target behavior is implemented +- [ ] Required verification actually ran (tests / lint / type-check) +- [ ] Evidence recorded in `feature_list.json` or `progress.md` +- [ ] Repository remains restartable from standard startup path + +## End of Session + +Before ending a session: + +1. Update `progress.md` with current state +2. Update `feature_list.json` with new feature status +3. Record any unresolved risks or blockers +4. Commit with descriptive message once work is in safe state +5. Leave repo clean enough for next session to run `./init.sh` immediately + +## Verification Commands + +```bash +# Full verification (recommended) +{{PRIMARY_VERIFICATION_COMMAND}} +``` + +Required checks: +{{VERIFICATION_COMMANDS}} + +## Escalation + +If you encounter: +- **Architecture decisions**: Consult project architecture docs if present, otherwise ask user +- **Unclear requirements**: Check product/requirements docs if present, otherwise ask user +- **Repeated test failures**: Update progress, flag for human review +- **Scope ambiguity**: Re-read `feature_list.json` for definition of done diff --git a/.agents/skills/harness-creator/templates/feature-list.json b/.agents/skills/harness-creator/templates/feature-list.json new file mode 100644 index 000000000..fff05b01c --- /dev/null +++ b/.agents/skills/harness-creator/templates/feature-list.json @@ -0,0 +1,44 @@ +{ + "features": [ + { + "id": "feat-001", + "name": "Project Setup", + "description": "Confirm the project can install dependencies, run verification, and start from a clean checkout", + "dependencies": [], + "status": "not-started", + "evidence": "" + }, + { + "id": "feat-002", + "name": "First User-Facing Feature", + "description": "Replace this placeholder with the first concrete behavior the agent should implement", + "dependencies": ["feat-001"], + "status": "not-started", + "evidence": "" + }, + { + "id": "feat-003", + "name": "Verification Coverage", + "description": "Add or confirm tests, type checks, linting, or manual verification for the active feature", + "dependencies": ["feat-002"], + "status": "not-started", + "evidence": "" + }, + { + "id": "feat-004", + "name": "Documentation Update", + "description": "Update README, architecture notes, or product docs affected by the implemented feature", + "dependencies": ["feat-003"], + "status": "not-started", + "evidence": "" + }, + { + "id": "feat-005", + "name": "Cleanup and Handoff", + "description": "Record verification evidence, update progress.md, and leave a clear next-session path", + "dependencies": ["feat-004"], + "status": "not-started", + "evidence": "" + } + ] +} diff --git a/.agents/skills/harness-creator/templates/feature-list.schema.json b/.agents/skills/harness-creator/templates/feature-list.schema.json new file mode 100644 index 000000000..45b852161 --- /dev/null +++ b/.agents/skills/harness-creator/templates/feature-list.schema.json @@ -0,0 +1,47 @@ +{ + "$schema": "http://json-schema.org/draft-07/schema#", + "title": "Feature List", + "description": "Track feature implementation state for agent-driven development", + "type": "object", + "properties": { + "features": { + "type": "array", + "items": { + "type": "object", + "properties": { + "id": { + "type": "string", + "description": "Unique feature identifier (e.g., feat-001)", + "pattern": "^feat-\\d+$" + }, + "name": { + "type": "string", + "description": "Short feature name" + }, + "description": { + "type": "string", + "description": "What this feature does" + }, + "dependencies": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Feature IDs that must be done before this one" + }, + "status": { + "type": "string", + "enum": ["not-started", "in-progress", "blocked", "done"], + "description": "Current implementation state" + }, + "evidence": { + "type": "string", + "description": "Verification evidence when status is 'done'" + } + }, + "required": ["id", "name", "description", "status"] + } + } + }, + "required": ["features"] +} diff --git a/.agents/skills/harness-creator/templates/init.sh b/.agents/skills/harness-creator/templates/init.sh new file mode 100644 index 000000000..a305a37f9 --- /dev/null +++ b/.agents/skills/harness-creator/templates/init.sh @@ -0,0 +1,75 @@ +#!/bin/bash +set -e + +echo "=== Harness Initialization ===" + +if [ -f package.json ]; then + if [ -f pnpm-lock.yaml ]; then + PM="pnpm" + elif [ -f yarn.lock ]; then + PM="yarn" + elif [ -f bun.lock ] || [ -f bun.lockb ]; then + PM="bun" + else + PM="npm" + fi + + echo "=== Installing dependencies with $PM ===" + if [ "$PM" = "npm" ]; then + npm install + else + "$PM" install + fi + + node -e "const s=require('./package.json').scripts||{}; process.exit(s.check||s.typecheck||s['type-check']?0:1)" && { + if node -e "const s=require('./package.json').scripts||{}; process.exit(s.check?0:1)"; then + [ "$PM" = "npm" ] && npm run check || "$PM" run check + elif node -e "const s=require('./package.json').scripts||{}; process.exit(s.typecheck?0:1)"; then + [ "$PM" = "npm" ] && npm run typecheck || "$PM" run typecheck + else + [ "$PM" = "npm" ] && npm run type-check || "$PM" run type-check + fi + } + + node -e "const s=require('./package.json').scripts||{}; process.exit(s.lint?0:1)" && { + [ "$PM" = "npm" ] && npm run lint || "$PM" run lint + } + + node -e "const s=require('./package.json').scripts||{}; process.exit(s.test?0:1)" && { + [ "$PM" = "npm" ] && npm test || "$PM" test + } + + node -e "const s=require('./package.json').scripts||{}; process.exit(s.build?0:1)" && { + [ "$PM" = "npm" ] && npm run build || "$PM" run build + } +elif [ -f pyproject.toml ] || [ -f requirements.txt ]; then + echo "=== Running Python verification ===" + python -m pytest + python -m compileall . +elif [ -f go.mod ]; then + echo "=== Running Go verification ===" + go test ./... +elif [ -f Cargo.toml ]; then + echo "=== Running Rust verification ===" + cargo test +elif [ -f pom.xml ]; then + echo "=== Running Maven verification ===" + mvn test +elif [ -f build.gradle ] || [ -f build.gradle.kts ]; then + echo "=== Running Gradle verification ===" + ./gradlew test +elif ls *.csproj *.sln >/dev/null 2>&1; then + echo "=== Running .NET verification ===" + dotnet test +else + echo "No recognized package manifest detected." + echo "Replace this section with the project's verification commands." +fi + +echo "=== Verification Complete ===" +echo "" +echo "Next steps:" +echo "1. Read feature_list.json to see current feature state" +echo "2. Pick ONE unfinished feature to work on" +echo "3. Implement only that feature" +echo "4. Re-run verification before claiming done" diff --git a/.agents/skills/harness-creator/templates/progress.md b/.agents/skills/harness-creator/templates/progress.md new file mode 100644 index 000000000..3710d5c35 --- /dev/null +++ b/.agents/skills/harness-creator/templates/progress.md @@ -0,0 +1,51 @@ +# Session Progress Log + +## Current State + +**Last Updated:** YYYY-MM-DD HH:MM +**Session ID:** [optional] +**Active Feature:** [feat-XXX - Feature Name] + +## Status + +### What's Done + +- [x] [Completed item 1] +- [x] [Completed item 2] + +### What's In Progress + +- [ ] [Current work item] + - Details: [specific task] + - Blockers: [if any] + +### What's Next + +1. [Next action item] +2. [Following action item] + +## Blockers / Risks + +- [ ] [Blocker 1]: [description, impact] +- [ ] [Risk 1]: [description, mitigation] + +## Decisions Made + +- **[Decision 1]**: [description] + - Context: [why this decision was made] + - Alternatives considered: [what else was discussed] + +## Files Modified This Session + +- `path/to/file1.ts` - [brief description of change] +- `path/to/file2.ts` - [brief description of change] + +## Evidence of Completion + +- [ ] Tests pass: `[command and output]` +- [ ] Type check clean: `[command and output]` +- [ ] Manual verification: `[what was tested]` + +## Notes for Next Session + +[Free-form notes that will help the next session pick up context] diff --git a/.agents/skills/harness-creator/templates/session-handoff.md b/.agents/skills/harness-creator/templates/session-handoff.md new file mode 100644 index 000000000..d9168cb72 --- /dev/null +++ b/.agents/skills/harness-creator/templates/session-handoff.md @@ -0,0 +1,40 @@ +# Session Handoff + +## Current Objective + +- Goal: +- Current status: +- Branch / commit: + +## Completed This Session + +- [ ] + +## Verification Evidence + +| Check | Command | Result | Notes | +|---|---|---|---| +| | | | | + +## Files Changed + +- + +## Decisions Made + +- + +## Blockers / Risks + +- + +## Next Session Startup + +1. Read `AGENTS.md`. +2. Read `feature_list.json` and `progress.md`. +3. Review this handoff. +4. Run `./init.sh` or the documented verification command before editing. + +## Recommended Next Step + +- diff --git a/AGENTS.md b/AGENTS.md index e1dc2115e..3fcc88e89 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,81 +1,60 @@ -# Open Deep Research 仓库概览 +# Repository Guidelines -## 项目描述 +## Startup Workflow -Open Deep Research 是一个可配置、完全开源的深度研究智能体(deep research agent)，支持多个模型提供商、搜索工具以及 MCP（Model Context Protocol，模型上下文协议）服务器。它能够通过并行处理自动执行研究任务，并生成内容全面的研究报告。 +Before writing code: -## 仓库结构 +1. Confirm the repository root and read `README.md`. +2. Read `feature_list.json`, `progress.md`, and any current `session-handoff.md`. +3. Select exactly one feature with an actionable status. +4. Run `./init.sh`. If the baseline fails, record the failure before changing code. +5. Review `git status --short` and preserve unrelated user changes. -### 根目录 +If dependencies are missing, create the development environment with `uv sync --extra dev`. -* `README.md` - 完整的项目文档，包含快速入门指南 -* `pyproject.toml` - Python 项目配置与依赖声明 -* `langgraph.json` - LangGraph 配置文件，用于定义主图(Graph)的入口点 -* `uv.lock` - UV 包管理器的依赖锁定文件 -* `LICENSE` - MIT 许可证 -* `.env.example` - 环境变量模板（不受版本控制追踪） +## Project Boundaries -### 核心实现（`src/open_deep_research/`） +Production code is in `src/open_deep_research/`; deployment authentication is in `src/security/`. Evaluation tooling is in `tests/`, while legacy code and its pytest suite are in `src/legacy/`. `deep_research_from_scratch/` is a separate tutorial package and must not be changed unless the active feature explicitly targets it. -* `deep_researcher.py` - LangGraph 的主要实现文件（入口点：`deep_researcher`） -* `configuration.py` - 配置管理与设置 -* `state.py` - 图状态(Graph state)定义与数据结构 -* `prompts.py` - 系统提示词与提示词模板 -* `utils.py` - 工具函数与辅助功能 -* `files/` - 研究输出文件与示例文件 +Work on one feature at a time. Stay in scope, do not rewrite unrelated files, and never commit `.env`, API keys, private MCP settings, or sensitive generated reports. -### 遗留实现（`src/legacy/`） +## State Artifacts -包含两种早期的研究实现： +- `feature_list.json` is the source of truth for feature status, dependencies, and evidence. +- `progress.md` records current state, decisions, blockers, and the next action. +- `session-handoff.md` provides a restart path for unfinished work. +- `init.sh` is the standard verification entrypoint. -* `graph.py` - 带有人机协同(human-in-the-loop)机制的规划与执行(plan-and-execute)工作流 -* `multi_agent.py` - 监督者—研究员(supervisor-researcher)多智能体架构 -* `legacy.md` - 遗留实现的说明文档 -* `CLAUDE.md` - 面向遗留实现的 Claude 专用指令 -* `tests/` - 遗留实现专用测试 +Add a concrete feature entry before starting new implementation work. Allowed status values are `not-started`, `in-progress`, `blocked`, and `completed`. -### 安全模块（`src/security/`） +## Verification Commands -* `auth.py` - 用于 LangGraph 部署的身份验证处理程序 +Run the full local baseline with: -### 测试（`tests/`） +```bash +./init.sh +``` -* `run_evaluate.py` - 主评估脚本，配置为在 Deep Research Bench 上运行 -* `evaluators.py` - 专用评估函数 -* `prompts.py` - 评估提示词与评估标准 -* `pairwise_evaluation.py` - 对比评估工具 -* `supervisor_parallel_evaluation.py` - 多线程并行评估 +It performs: -### 示例（`examples/`） +- `python -m compileall -q src` +- `python -m ruff check .` +- `python -m mypy src` +- `python -m pytest --collect-only -q src/legacy/tests` -* `arxiv.md` - ArXiv 研究示例 -* `pubmed.md` - PubMed 研究示例 -* `inference-market.md` - 推理市场分析示例 +The comprehensive evaluation, `python tests/run_evaluate.py`, uses external services and may incur cost. Run it only when explicitly required and credentials are available. -### 手把手教学（deep_research_from_scratch/） -* 改文件夹下放的该项目进面向人类初学者的教学jupyter notebook内容，可视为与本项目的实际运行和使用完全解耦的模块 +## Definition of Done -## 核心技术 +A feature is done only when its scoped behavior is complete, relevant verification has run, evidence is recorded, documentation is updated when necessary, and the repository has a clear restart path. -* **LangGraph** - 工作流编排与图执行(Graph execution) -* **LangChain** - 大语言模型集成与工具调用(tool calling) -* **多个 LLM 提供商** - 支持 OpenAI、Anthropic、Google、Groq 和 DeepSeek -* **搜索 API** - 支持 Tavily、OpenAI/Anthropic 原生搜索、DuckDuckGo 和 Exa -* **MCP Servers** - 通过 Model Context Protocol 扩展智能体能力 +## End of Session -## 开发命令 +Before ending: -* `uvx langgraph dev` - 启动带有 LangGraph Studio 的开发服务器 -* `python tests/run_evaluate.py` - 运行完整评估 -* `ruff check` - 执行代码 lint 检查 -* `mypy` - 执行类型检查 +1. Update the active feature status and evidence. +2. Update `progress.md` with files changed, verification results, blockers, and the recommended next step. +3. Refresh `session-handoff.md` when work remains unfinished. +4. Check `git status --short` and leave unrelated changes untouched. -## 配置 - -所有设置均可通过以下方式进行配置： - -* 环境变量（`.env` 文件） -* LangGraph Studio 中的 Web UI -* 直接修改配置文件 - -关键设置包括模型选择、搜索 API 选择、并发限制以及 MCP server 配置。 +Leave a clean restart path: the next session must recover the active objective and rerun verification from these files without relying on chat history. diff --git a/AGENTS.override.md b/AGENTS.override.md new file mode 100644 index 000000000..8690221c1 --- /dev/null +++ b/AGENTS.override.md @@ -0,0 +1,81 @@ +# Open Deep Research 仓库概览 + +## 项目描述 + +Open Deep Research 是一个可配置、完全开源的深度研究智能体(deep research agent)，支持多个模型提供商、搜索工具以及 MCP（Model Context Protocol，模型上下文协议）服务器。它能够通过并行处理自动执行研究任务，并生成内容全面的研究报告。 + +## 仓库结构 + +### 根目录 + +* `README.md` - 完整的项目文档，包含快速入门指南 +* `pyproject.toml` - Python 项目配置与依赖声明 +* `langgraph.json` - LangGraph 配置文件，用于定义主图(Graph)的入口点 +* `uv.lock` - UV 包管理器的依赖锁定文件 +* `LICENSE` - MIT 许可证 +* `.env.example` - 环境变量模板（不受版本控制追踪） + +### 核心实现（`src/open_deep_research/`） + +* `deep_researcher.py` - LangGraph 的主要实现文件（入口点：`deep_researcher`） +* `configuration.py` - 配置管理与设置 +* `state.py` - 图状态(Graph state)定义与数据结构 +* `prompts.py` - 系统提示词与提示词模板 +* `utils.py` - 工具函数与辅助功能 +* `files/` - 研究输出文件与示例文件 + +### 遗留实现（`src/legacy/`） + +包含两种早期的研究实现： + +* `graph.py` - 带有人机协同(human-in-the-loop)机制的规划与执行(plan-and-execute)工作流 +* `multi_agent.py` - 监督者—研究员(supervisor-researcher)多智能体架构 +* `legacy.md` - 遗留实现的说明文档 +* `CLAUDE.md` - 面向遗留实现的 Claude 专用指令 +* `tests/` - 遗留实现专用测试 + +### 安全模块（`src/security/`） + +* `auth.py` - 用于 LangGraph 部署的身份验证处理程序 + +### 测试（`tests/`） + +* `run_evaluate.py` - 主评估脚本，配置为在 Deep Research Bench 上运行 +* `evaluators.py` - 专用评估函数 +* `prompts.py` - 评估提示词与评估标准 +* `pairwise_evaluation.py` - 对比评估工具 +* `supervisor_parallel_evaluation.py` - 多线程并行评估 + +### 示例（`examples/`） + +* `arxiv.md` - ArXiv 研究示例 +* `pubmed.md` - PubMed 研究示例 +* `inference-market.md` - 推理市场分析示例 + +### 手把手教学（deep_research_from_scratch/） +* 改文件夹下放的该项目进面向人类初学者的教学jupyter notebook内容，可视为与本项目的实际运行和使用完全解耦的模块 + +## 核心技术 + +* **LangGraph** - 工作流编排与图执行(Graph execution) +* **LangChain** - 大语言模型集成与工具调用(tool calling) +* **多个 LLM 提供商** - 支持 OpenAI、Anthropic、Google、Groq 和 DeepSeek +* **搜索 API** - 支持 Tavily、OpenAI/Anthropic 原生搜索、DuckDuckGo 和 Exa +* **MCP Servers** - 通过 Model Context Protocol 扩展智能体能力 + +## 开发命令 +* `conda activate open-deep-research`启动本项目的专用环境 +* `langgraph dev --allow-blocking `使用langgraph studio UI，启动带有 LangGraph Studio 的开发服务器 +* `python tests/run_evaluate.py` - 运行完整评估 +* `ruff check` - 执行代码 lint 检查 +* `mypy` - 执行类型检查 + +## 配置 + +所有设置均可通过以下方式进行配置： + +* 环境变量（`.env` 文件） +* LangGraph Studio 中的 Web UI +* 直接修改配置文件 + +关键设置包括模型选择、搜索 API 选择、并发限制以及 MCP server 配置。 diff --git a/README.md b/README.md index 256e0adbe..ce551510f 100644 --- a/README.md +++ b/README.md @@ -21,12 +21,6 @@ Deep research has broken out as one of the most popular agent applications. This ### 🚀 Quickstart 1. Clone the repository and activate a virtual environment: -```bash -git clone https://github.com/langchain-ai/open_deep_research.git -cd open_deep_research -uv venv -source .venv/bin/activate # On Windows: .venv\Scripts\activate -``` ```bash git clone https://github.com/langchain-ai/open_deep_research.git @@ -38,11 +32,6 @@ python -m pip install --upgrade pip setuptools wheel ``` 2. Install dependencies: -```bash -uv sync -# or -uv pip install -r pyproject.toml -``` ```bash pip install -e . # 可编辑模式安装当前目录中的项目，并基于pyproject.toml进行依赖解析 @@ -56,8 +45,7 @@ cp .env.example .env 4. Launch agent with the LangGraph server locally: ```bash -# Install dependencies and start the LangGraph server -uvx --refresh --from "langgraph-cli[inmem]" --with-editable . --python 3.11 langgraph dev --allow-blocking +langgraph dev --allow-blocking ``` This will open the LangGraph Studio UI in your browser. diff --git a/feature_list.json b/feature_list.json new file mode 100644 index 000000000..41546282d --- /dev/null +++ b/feature_list.json @@ -0,0 +1,12 @@ +{ + "features": [ + { + "id": "harness-001", + "name": "Initialize minimal coding-agent harness", + "description": "Add repository instructions, persistent feature state, local verification, and a session handoff path without changing business code.", + "dependencies": [], + "status": "completed", + "evidence": "Five minimal harness artifacts created without business-code changes; harness validator passed 25/25 structural checks (100/100)." + } + ] +} diff --git a/init.sh b/init.sh new file mode 100644 index 000000000..a4d894998 --- /dev/null +++ b/init.sh @@ -0,0 +1,19 @@ +#!/bin/bash +set -e + +echo "=== Open Deep Research Harness Verification ===" + +echo "=== Compile Python sources ===" +python -m compileall -q src + +echo "=== Ruff ===" +python -m ruff check . + +echo "=== Mypy ===" +python -m mypy src + +echo "=== Collect legacy tests without external API execution ===" +python -m pytest --collect-only -q src/legacy/tests + +echo "=== Verification Complete ===" +echo "Read feature_list.json, select one feature, and record evidence in progress.md." diff --git a/progress.md b/progress.md new file mode 100644 index 000000000..64316a8b8 --- /dev/null +++ b/progress.md @@ -0,0 +1,52 @@ +# Session Progress Log + +## Current State + +**Last Updated:** 2026-06-21 +**Active Feature:** `harness-001` — Initialize minimal coding-agent harness +**Status:** Completed + +## What's Done + +- Added the five minimal harness artifacts. +- Restricted routine verification to local, non-billable checks. +- Preserved existing business code and unrelated working-tree changes. + +## Final Validation + +- Passed all 25 structural harness checks. +- Confirmed `feature_list.json` parses successfully and no business code was changed. + +## What's Next + +1. Add a new feature entry before beginning business-code work. +2. Run `uv sync --extra dev` if the development environment is unavailable. +3. Run `./init.sh` and record the result. + +## Blockers / Risks + +- The current shell may not expose project tools on `PATH`; use the environment created by `uv sync --extra dev`. +- Legacy report-quality tests invoke external services, so `init.sh` only collects them. + +## Decisions Made + +- `tests/run_evaluate.py` is excluded from routine startup because it requires credentials and may incur API cost. +- Harness state is stored in versioned files rather than chat history. + +## Files Modified This Session + +- `AGENTS.md` +- `feature_list.json` +- `progress.md` +- `init.sh` +- `session-handoff.md` + +## Verification Evidence + +- Structural validation: `100/100` (`25/25` checks passed) +- Feature tracker JSON parsing: passed +- Business-code tests: not required; no business code changed + +## Notes for Next Session + +Read the state files, add one concrete feature, and run `./init.sh` before implementation. diff --git a/session-handoff.md b/session-handoff.md new file mode 100644 index 000000000..704a4b189 --- /dev/null +++ b/session-handoff.md @@ -0,0 +1,45 @@ +# Session Handoff + +## Current Objective + +- Goal: Initialize and validate the repository's minimal coding-agent harness. +- Current status: Completed; all structural checks passed. +- Branch / commit: Use the current working tree; no commit created. + +## Completed This Session + +- Added instructions, feature state, progress tracking, verification, and lifecycle handoff files. +- Kept business code unchanged. + +## Verification Evidence + +| Check | Command | Result | Notes | +|---|---|---|---| +| Structural validation | `validate-harness.mjs --target .` | Passed | 100/100; 25/25 checks | +| Feature tracker | PowerShell JSON parse | Passed | Valid JSON | + +## Files Changed + +- `AGENTS.md` +- `feature_list.json` +- `progress.md` +- `init.sh` +- `session-handoff.md` + +## Decisions Made + +- Routine verification does not execute external model or search APIs. + +## Blockers / Risks + +- Project development tools require the configured Python environment; run `uv sync --extra dev` if needed. + +## Next Session Startup + +1. Read `AGENTS.md`, `feature_list.json`, and `progress.md`. +2. Review this handoff and `git status --short`. +3. Run `./init.sh` before editing. + +## Recommended Next Step + +- Add one concrete feature to `feature_list.json` before changing business code. diff --git a/skills-lock.json b/skills-lock.json new file mode 100644 index 000000000..033d35147 --- /dev/null +++ b/skills-lock.json @@ -0,0 +1,11 @@ +{ + "version": 1, + "skills": { + "harness-creator": { + "source": "walkinglabs/learn-harness-engineering", + "sourceType": "github", + "skillPath": "skills/harness-creator/SKILL.md", + "computedHash": "399b47cba9b61c43deedc165fe8bab8770da436a389de425b248fa8825a4fec3" + } + } +} From c32774f716dc77c85af09c58a84f73106b395dc5 Mon Sep 17 00:00:00 2001 From: magic-ZSS <1165626853@qq.com> Date: Sun, 21 Jun 2026 02:21:56 +0800 Subject: [PATCH 3/5] =?UTF-8?q?init=20$harness-creator=E4=B8=BA=E5=BD=93?= =?UTF-8?q?=E5=89=8D=E4=BB=93=E5=BA=93=E5=88=9D=E5=A7=8B=E5=8C=96=E6=9C=80?= =?UTF-8?q?=E5=B0=8F=E5=8F=AF=E7=94=A8=20Harness?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This reverts commit 7e54c115035f8a7a1ae041300384ea354724d290. --- .agents/skills/harness-creator/README.md | 83 ------ .agents/skills/harness-creator/SKILL.md | 107 ------- .agents/skills/harness-creator/SKILL.md.en | 107 ------- .../skills/harness-creator/agents/openai.yaml | 7 - .../skills/harness-creator/evals/evals.json | 136 --------- .../references/context-engineering-pattern.md | 150 ---------- .../harness-creator/references/gotchas.md | 209 -------------- .../references/lifecycle-bootstrap-pattern.md | 264 ------------------ .../references/memory-persistence-pattern.md | 110 -------- .../references/multi-agent-pattern.md | 193 ------------- .../references/skill-runtime-pattern.md | 43 --- .../references/tool-registry-pattern.md | 200 ------------- .../scripts/create-harness.mjs | 75 ----- .../scripts/render-assessment-html.mjs | 26 -- .../harness-creator/scripts/run-benchmark.mjs | 121 -------- .../scripts/validate-harness.mjs | 43 --- .../harness-creator/templates/agents.md | 68 ----- .../templates/feature-list.json | 44 --- .../templates/feature-list.schema.json | 47 ---- .../skills/harness-creator/templates/init.sh | 75 ----- .../harness-creator/templates/progress.md | 51 ---- .../templates/session-handoff.md | 40 --- AGENTS.md | 97 ++++--- AGENTS.override.md | 81 ------ README.md | 14 +- feature_list.json | 12 - init.sh | 19 -- progress.md | 52 ---- session-handoff.md | 45 --- skills-lock.json | 11 - 30 files changed, 72 insertions(+), 2458 deletions(-) delete mode 100644 .agents/skills/harness-creator/README.md delete mode 100644 .agents/skills/harness-creator/SKILL.md delete mode 100644 .agents/skills/harness-creator/SKILL.md.en delete mode 100644 .agents/skills/harness-creator/agents/openai.yaml delete mode 100644 .agents/skills/harness-creator/evals/evals.json delete mode 100644 .agents/skills/harness-creator/references/context-engineering-pattern.md delete mode 100644 .agents/skills/harness-creator/references/gotchas.md delete mode 100644 .agents/skills/harness-creator/references/lifecycle-bootstrap-pattern.md delete mode 100644 .agents/skills/harness-creator/references/memory-persistence-pattern.md delete mode 100644 .agents/skills/harness-creator/references/multi-agent-pattern.md delete mode 100644 .agents/skills/harness-creator/references/skill-runtime-pattern.md delete mode 100644 .agents/skills/harness-creator/references/tool-registry-pattern.md delete mode 100644 .agents/skills/harness-creator/scripts/create-harness.mjs delete mode 100644 .agents/skills/harness-creator/scripts/render-assessment-html.mjs delete mode 100644 .agents/skills/harness-creator/scripts/run-benchmark.mjs delete mode 100644 .agents/skills/harness-creator/scripts/validate-harness.mjs delete mode 100644 .agents/skills/harness-creator/templates/agents.md delete mode 100644 .agents/skills/harness-creator/templates/feature-list.json delete mode 100644 .agents/skills/harness-creator/templates/feature-list.schema.json delete mode 100644 .agents/skills/harness-creator/templates/init.sh delete mode 100644 .agents/skills/harness-creator/templates/progress.md delete mode 100644 .agents/skills/harness-creator/templates/session-handoff.md delete mode 100644 AGENTS.override.md delete mode 100644 feature_list.json delete mode 100644 init.sh delete mode 100644 progress.md delete mode 100644 session-handoff.md delete mode 100644 skills-lock.json diff --git a/.agents/skills/harness-creator/README.md b/.agents/skills/harness-creator/README.md deleted file mode 100644 index 862766338..000000000 --- a/.agents/skills/harness-creator/README.md +++ /dev/null @@ -1,83 +0,0 @@ -# harness-creator - -A compact skill for building and auditing harnesses around AI coding agents. - -It helps a repository provide five things agents need: instructions, state, verification, scope boundaries, and lifecycle handoff. - -## Install - -```bash -npx skills add walkinglabs/learn-harness-engineering --skill harness-creator -``` - -Or copy `skills/harness-creator/` into your skill path. - -## Use - -```bash -node skills/harness-creator/scripts/create-harness.mjs --target /path/to/project -node skills/harness-creator/scripts/validate-harness.mjs --target /path/to/project -node skills/harness-creator/scripts/run-benchmark.mjs --target /path/to/project --html /path/to/report.html -``` - -The scripts use only Node.js built-in modules. They can be run after copying the skill directory into another repository. - -## What It Creates - -- `AGENTS.md` or `CLAUDE.md` -- `feature_list.json` -- `progress.md` -- `init.sh` -- `session-handoff.md` - -`create-harness.mjs` detects common project types and package managers. It supports Node/npm/pnpm/yarn/bun, Python, Go, Rust, Maven, Gradle, and .NET at a basic verification-command level. - -## What It Checks - -`validate-harness.mjs` scores the five harness subsystems: - -1. Instructions -2. State -3. Verification -4. Scope -5. Lifecycle - -The score is structural. It tells you whether the harness is present and coherent; it does not replace real before/after agent-session testing. - -## Status - -- [x] Minimal harness scaffolding -- [x] Five-subsystem validation -- [x] HTML assessment report -- [x] Structural benchmark report -- [x] 10 eval cases -- [x] Generic verification detection for common stacks -- [ ] Optional real before/after agent-session replay - -## Files - -```text -harness-creator/ -├── SKILL.md -├── metadata.json -├── agents/openai.yaml -├── scripts/ -│ ├── create-harness.mjs -│ ├── validate-harness.mjs -│ ├── render-assessment-html.mjs -│ ├── run-benchmark.mjs -│ └── lib/harness-utils.mjs -├── templates/ -│ ├── agents.md -│ ├── feature-list.json -│ ├── feature-list.schema.json -│ ├── init.sh -│ ├── progress.md -│ └── session-handoff.md -├── references/ -└── evals/evals.json -``` - -## Boundaries - -This skill is for harness engineering, not model selection, prompt tuning alone, or app architecture. Keep project-specific facts in the target repository. diff --git a/.agents/skills/harness-creator/SKILL.md b/.agents/skills/harness-creator/SKILL.md deleted file mode 100644 index 802fed82a..000000000 --- a/.agents/skills/harness-creator/SKILL.md +++ /dev/null @@ -1,107 +0,0 @@ ---- -name: harness-creator -description: >- - Build, audit, and improve lightweight harnesses for AI coding agents: AGENTS.md/CLAUDE.md, - feature state, verification workflows, scope boundaries, lifecycle handoff, - memory persistence, context control, tool safety, and multi-agent coordination. -license: MIT ---- - -# Harness Creator - -Use this skill to make a repository easier for coding agents to start, stay in scope, verify work, and resume across sessions. Keep the harness small enough that agents actually follow it. - -Not for model selection, prompt tuning in isolation, chat UI design, or general app architecture. - -## Core Model - -Every useful coding-agent harness has five subsystems: - -| Subsystem | Minimal artifact | Purpose | -|---|---|---| -| Instructions | `AGENTS.md` or `CLAUDE.md` | Startup path, working rules, definition of done | -| State | `feature_list.json`, `progress.md` | Current feature, status, evidence, next step | -| Verification | `init.sh` or documented commands | Tests/checks the agent must run before claiming done | -| Scope | Feature dependencies and done criteria | Prevents overreach and half-finished work | -| Lifecycle | `session-handoff.md`, end-of-session routine | Makes the next session restartable | - -## First Move - -1. Inspect what already exists: instruction files, feature/state files, verification commands, docs, package manifests. -2. Ask only for missing context that cannot be inferred safely: target agent, desired file name, tolerance for structure, and whether overwriting is allowed. -3. Prefer a minimal harness first. Add memory, tool safety, multi-agent, or benchmark details only when the user's problem calls for them. - -## Common Tasks - -### Create a harness - -Use the bundled script when working on a local repository: - -```bash -node skills/harness-creator/scripts/create-harness.mjs --target /path/to/project -``` - -Options: - -- `--agent-file CLAUDE.md` for Claude-oriented projects. -- `--package-manager npm|pnpm|yarn|bun` when detection is wrong. -- `--commands "cmd one,cmd two"` for custom verification. -- `--force` only after confirming overwrites are acceptable. - -Then explain what was created and how the user should replace placeholder feature entries. - -### Audit an existing harness - -Run: - -```bash -node skills/harness-creator/scripts/validate-harness.mjs --target /path/to/project -``` - -Report the five subsystem scores, the lowest-scoring area, and the first 2-3 changes that would improve reliability. Treat the lowest score as a candidate bottleneck; confirm with failures, logs, or task outcomes before claiming causality. - -### Produce a report - -Use when the user wants a shareable assessment: - -```bash -node skills/harness-creator/scripts/render-assessment-html.mjs --target /path/to/project -node skills/harness-creator/scripts/run-benchmark.mjs --target /path/to/project --html /path/to/report.html -``` - -Be clear that this is a structural benchmark. Real effectiveness still needs before/after agent sessions on representative tasks. - -## When to Read References - -Load only the reference needed for the user's problem: - -- Memory across sessions: [Memory Persistence](references/memory-persistence-pattern.md) -- Reusable workflows as skills: [Skill Runtime](references/skill-runtime-pattern.md) -- Permissions, tools, concurrency: [Tool Registry & Safety](references/tool-registry-pattern.md) -- Context budget and progressive disclosure: [Context Engineering](references/context-engineering-pattern.md) -- Delegation and parallel agents: [Multi-Agent Coordination](references/multi-agent-pattern.md) -- Hooks, startup, long-running work: [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) -- Non-obvious failure modes: [Gotchas](references/gotchas.md) - -## Design Rules - -- Keep the root instruction file short: routing and invariants, not a full manual. -- Put project facts in project docs, not in the skill. -- Make verification commands explicit and runnable. -- Require evidence before marking a feature done. -- Use one active feature unless the harness has explicit multi-agent ownership boundaries. -- Prefer append/update state files over relying on chat history. -- Never hide destructive behavior in scripts; overwrites require explicit user approval. - -## Deliverable Checklist - -For a usable minimal harness, leave the target project with: - -- [ ] `AGENTS.md` or `CLAUDE.md` -- [ ] `feature_list.json` -- [ ] `progress.md` -- [ ] `init.sh` -- [ ] Optional `session-handoff.md` for multi-session work -- [ ] Documented verification evidence or next action - -If you cannot create files, provide exact file contents and commands instead. diff --git a/.agents/skills/harness-creator/SKILL.md.en b/.agents/skills/harness-creator/SKILL.md.en deleted file mode 100644 index 802fed82a..000000000 --- a/.agents/skills/harness-creator/SKILL.md.en +++ /dev/null @@ -1,107 +0,0 @@ ---- -name: harness-creator -description: >- - Build, audit, and improve lightweight harnesses for AI coding agents: AGENTS.md/CLAUDE.md, - feature state, verification workflows, scope boundaries, lifecycle handoff, - memory persistence, context control, tool safety, and multi-agent coordination. -license: MIT ---- - -# Harness Creator - -Use this skill to make a repository easier for coding agents to start, stay in scope, verify work, and resume across sessions. Keep the harness small enough that agents actually follow it. - -Not for model selection, prompt tuning in isolation, chat UI design, or general app architecture. - -## Core Model - -Every useful coding-agent harness has five subsystems: - -| Subsystem | Minimal artifact | Purpose | -|---|---|---| -| Instructions | `AGENTS.md` or `CLAUDE.md` | Startup path, working rules, definition of done | -| State | `feature_list.json`, `progress.md` | Current feature, status, evidence, next step | -| Verification | `init.sh` or documented commands | Tests/checks the agent must run before claiming done | -| Scope | Feature dependencies and done criteria | Prevents overreach and half-finished work | -| Lifecycle | `session-handoff.md`, end-of-session routine | Makes the next session restartable | - -## First Move - -1. Inspect what already exists: instruction files, feature/state files, verification commands, docs, package manifests. -2. Ask only for missing context that cannot be inferred safely: target agent, desired file name, tolerance for structure, and whether overwriting is allowed. -3. Prefer a minimal harness first. Add memory, tool safety, multi-agent, or benchmark details only when the user's problem calls for them. - -## Common Tasks - -### Create a harness - -Use the bundled script when working on a local repository: - -```bash -node skills/harness-creator/scripts/create-harness.mjs --target /path/to/project -``` - -Options: - -- `--agent-file CLAUDE.md` for Claude-oriented projects. -- `--package-manager npm|pnpm|yarn|bun` when detection is wrong. -- `--commands "cmd one,cmd two"` for custom verification. -- `--force` only after confirming overwrites are acceptable. - -Then explain what was created and how the user should replace placeholder feature entries. - -### Audit an existing harness - -Run: - -```bash -node skills/harness-creator/scripts/validate-harness.mjs --target /path/to/project -``` - -Report the five subsystem scores, the lowest-scoring area, and the first 2-3 changes that would improve reliability. Treat the lowest score as a candidate bottleneck; confirm with failures, logs, or task outcomes before claiming causality. - -### Produce a report - -Use when the user wants a shareable assessment: - -```bash -node skills/harness-creator/scripts/render-assessment-html.mjs --target /path/to/project -node skills/harness-creator/scripts/run-benchmark.mjs --target /path/to/project --html /path/to/report.html -``` - -Be clear that this is a structural benchmark. Real effectiveness still needs before/after agent sessions on representative tasks. - -## When to Read References - -Load only the reference needed for the user's problem: - -- Memory across sessions: [Memory Persistence](references/memory-persistence-pattern.md) -- Reusable workflows as skills: [Skill Runtime](references/skill-runtime-pattern.md) -- Permissions, tools, concurrency: [Tool Registry & Safety](references/tool-registry-pattern.md) -- Context budget and progressive disclosure: [Context Engineering](references/context-engineering-pattern.md) -- Delegation and parallel agents: [Multi-Agent Coordination](references/multi-agent-pattern.md) -- Hooks, startup, long-running work: [Lifecycle & Bootstrap](references/lifecycle-bootstrap-pattern.md) -- Non-obvious failure modes: [Gotchas](references/gotchas.md) - -## Design Rules - -- Keep the root instruction file short: routing and invariants, not a full manual. -- Put project facts in project docs, not in the skill. -- Make verification commands explicit and runnable. -- Require evidence before marking a feature done. -- Use one active feature unless the harness has explicit multi-agent ownership boundaries. -- Prefer append/update state files over relying on chat history. -- Never hide destructive behavior in scripts; overwrites require explicit user approval. - -## Deliverable Checklist - -For a usable minimal harness, leave the target project with: - -- [ ] `AGENTS.md` or `CLAUDE.md` -- [ ] `feature_list.json` -- [ ] `progress.md` -- [ ] `init.sh` -- [ ] Optional `session-handoff.md` for multi-session work -- [ ] Documented verification evidence or next action - -If you cannot create files, provide exact file contents and commands instead. diff --git a/.agents/skills/harness-creator/agents/openai.yaml b/.agents/skills/harness-creator/agents/openai.yaml deleted file mode 100644 index 500d7ae09..000000000 --- a/.agents/skills/harness-creator/agents/openai.yaml +++ /dev/null @@ -1,7 +0,0 @@ -interface: - display_name: "Harness Creator" - short_description: "Scaffold and audit coding-agent harnesses" - default_prompt: "Use $harness-creator to create, validate, and benchmark a production-ready coding-agent harness for this repository." - -policy: - allow_implicit_invocation: true diff --git a/.agents/skills/harness-creator/evals/evals.json b/.agents/skills/harness-creator/evals/evals.json deleted file mode 100644 index 723d5ee68..000000000 --- a/.agents/skills/harness-creator/evals/evals.json +++ /dev/null @@ -1,136 +0,0 @@ -{ - "skill_name": "harness-creator", - "evals": [ - { - "id": 1, - "name": "Minimal Harness Creation", - "prompt": "I have a new TypeScript + React project with no agent setup. Create a minimal harness that makes my agent reliable for single-feature development.", - "expected_output": "AGENTS.md (~50-100 lines), feature_list.json with 3-5 placeholder features, init.sh with verification commands", - "files": [], - "expectations": [ - "AGENTS.md includes startup workflow (read files, run init, check feature list)", - "AGENTS.md includes one-feature-at-a-time policy", - "feature_list.json has valid JSON with id, name, description, status fields", - "init.sh runs install, type/static check, test, and build commands when available", - "All files are in project root directory" - ] - }, - { - "id": 2, - "name": "Session Continuity Setup", - "prompt": "My agent forgets everything between sessions. I need persistent memory and session handoff so it can work on multi-day features.", - "expected_output": "progress.md template, session-handoff.md structure, memory directory setup instructions", - "files": [], - "expectations": [ - "progress.md includes sections: current state, what's done, what's in progress, blockers, next session should", - "session-handoff.md includes: what was accomplished, what remains, blockers/decisions, files modified", - "Instructions for memory directory creation (.claude/memory/ or similar)", - "Two-step save invariant explained (topic file then index)" - ] - }, - { - "id": 3, - "name": "Harness Assessment", - "prompt": "I have an existing AGENTS.md file but my agent still breaks things. Assess my harness and tell me what to improve first.", - "expected_output": "Five-subsystem assessment with scores 1-5 for each, bottleneck identified, prioritized improvement plan", - "files": ["existing-AGENTS.md"], - "expectations": [ - "Assessment covers all 5 subsystems: Instructions, State, Verification, Scope, Lifecycle", - "Each subsystem scored 1-5 with justification", - "Lowest-scoring subsystem identified as bottleneck or candidate bottleneck", - "Prioritized improvement plan with 2-3 concrete next steps" - ] - }, - { - "id": 4, - "name": "Verification Workflow Design", - "prompt": "My agent says 'done' but the tests fail. Design a verification workflow that forces the agent to actually verify before claiming completion.", - "expected_output": "Verification commands list, AGENTS.md section updates, optional quality score tracking", - "files": [], - "expectations": [ - "Explicit verification commands listed (tests, lint, type-check, build)", - "AGENTS.md includes Definition of Done section with verification requirement", - "End-of-session checklist includes verification evidence recording", - "Failure path says not to claim done when verification fails" - ] - }, - { - "id": 5, - "name": "Memory Taxonomy Design", - "prompt": "I want my agent to remember project conventions and user preferences across sessions. Design a memory taxonomy and tell me what belongs where.", - "expected_output": "Memory layer definitions, type taxonomy for auto-memory, what to save vs what to skip", - "files": [], - "expectations": [ - "Instruction memory defined (human-curated, version-controlled)", - "Auto-memory defined (agent-written, persistent)", - "Type taxonomy with 3-4 types (e.g., user/feedback/project/reference)", - "Clear guidance on what NOT to save (derivable content)" - ] - }, - { - "id": 6, - "name": "Tool Safety Design", - "prompt": "My custom coding agent can run shell commands and edit files. Design the tool registry and permission harness so it is powerful but does not silently do dangerous things.", - "expected_output": "Tool registry policy with per-call concurrency classification, default permissions, denial handling, and audit trail", - "files": [], - "expectations": [ - "Sensitive tools default to ask or deny instead of allow", - "Concurrency safety is classified per call rather than per tool", - "Permission evaluation side effects are acknowledged and not cached unsafely", - "Audit trail records command, decision, and reason" - ] - }, - { - "id": 7, - "name": "Context Budget Plan", - "prompt": "My agent loads too many docs at startup and gets slow. Design a context strategy that keeps important context while controlling token cost.", - "expected_output": "Progressive disclosure plan using select/write/compress/isolate operations and explicit context budgets", - "files": [], - "expectations": [ - "Defines always-loaded metadata versus on-demand references", - "Uses SELECT, WRITE, COMPRESS, and ISOLATE operations", - "Includes hard caps or budget thresholds", - "Explains invalidation for memoized context builders" - ] - }, - { - "id": 8, - "name": "Multi-Agent Coordination", - "prompt": "I want multiple agents to work on a large refactor without stepping on each other. Design a multi-agent harness and rules for delegation.", - "expected_output": "Coordinator/delegation design with ownership boundaries, context sharing policy, and merge/review gates", - "files": [], - "expectations": [ - "Defines coordinator, worker, and optional reviewer responsibilities", - "Assigns disjoint file or module ownership", - "Prevents recursive fork children from forking again", - "Includes integration and verification gate before claiming done" - ] - }, - { - "id": 9, - "name": "Lifecycle Bootstrap", - "prompt": "Every new session starts inconsistently. Design startup, hook, and handoff lifecycle rules so a new agent can resume safely.", - "expected_output": "Lifecycle bootstrap design with init.sh, clean-state checks, handoff reads, and hook trust boundaries", - "files": [], - "expectations": [ - "init.sh is the standard startup and verification entrypoint", - "Startup reads AGENTS.md, feature_list.json, progress.md, and handoff when present", - "End-of-session procedure records evidence, blockers, and next step", - "Hooks are gated by trust and failure behavior is explicit" - ] - }, - { - "id": 10, - "name": "Scripted Harness Validation", - "prompt": "Use the harness-creator scripts to scaffold a harness, validate it, and produce an HTML assessment report for a repository.", - "expected_output": "Commands using create-harness.mjs, validate-harness.mjs, and render-assessment-html.mjs plus interpretation of score", - "files": [], - "expectations": [ - "Uses create-harness.mjs with a target directory", - "Uses validate-harness.mjs and explains five-subsystem score output", - "Uses render-assessment-html.mjs or run-benchmark.mjs with --html", - "Explains that structural benchmark complements but does not replace real before/after agent sessions" - ] - } - ] -} diff --git a/.agents/skills/harness-creator/references/context-engineering-pattern.md b/.agents/skills/harness-creator/references/context-engineering-pattern.md deleted file mode 100644 index 6bf6e3d83..000000000 --- a/.agents/skills/harness-creator/references/context-engineering-pattern.md +++ /dev/null @@ -1,150 +0,0 @@ -# Context Engineering Pattern - -## Problem - -Agents fail when context is managed poorly: -- **Too much context** → Session startup is slow, token costs explode, model gets lost in details -- **Too little context** → Agent makes wrong assumptions, reinvents wheels, violates conventions -- **Wrong context** → Agent focuses on low-level details, misses architectural constraints - -Context is not a dump. It's a budget that must be managed with explicit operations. - -## Golden Rules - -### Four Context Operations - -Every token in the window should earn its place through one of four operations: - -1. **SELECT** — Load context just-in-time, not all-at-once -2. **WRITE** — Agent writes back to persistent storage (memory, state, rules) -3. **COMPRESS** — Reactive compaction of older turns mid-session -4. **ISOLATE** — Delegated work must not pollute parent context - -### Progressive Disclosure - -Three-tier loading: - -``` -Tier 1: Metadata (always present, cheap) - → Feature list, memory index, session status - -Tier 2: Instructions (loaded on activation) - → AGENTS.md, skill bodies, style guides - -Tier 3: Resources (loaded on demand) - → Architecture docs, API references, examples -``` - -### Memoize Expensive Builders, Invalidate Explicitly - -Context builders (e.g., "load all recent git commits") should be memoized to avoid redundant work, but **must** be invalidated at known mutation points — not reactively. Every mutation point must clear its corresponding cache. - -## When To Use - -- Agent performance degrades in long sessions -- Startup is slow due to eager context loading -- Delegated work pollutes the parent context -- Token costs are unpredictable - -## Tradeoffs - -| Decision | Benefit | Cost | -|---|---|---| -| JIT loading | Fast startup, low idle cost | Agent can't reason about skills until activated | -| Hard caps per block | Predictable token budget | May truncate useful context | -| Manual cache invalidation | No reactive staleness | Developer must add invalidation at each mutation | -| Isolation for delegation | Clean parent context | Child can't see parent's accumulated context | - -## Implementation Patterns - -### Select Pattern - -```markdown -## Startup Context (Loaded Immediately) - -- Repository root path -- Tech stack (one line) -- Active feature ID from feature_list.json - -## On-Demand Context (Loaded When Triggered) - -- Skill: Read when skill activates -- Architecture docs: Read when implementing new feature -- API reference: Read when calling external services -``` - -**Key moves:** -- Audit current context cost per turn -- Apply hard caps to every variable-length block -- Add truncation recovery pointers ("call list_files for full output") - -### Compress Pattern - -Long sessions exhaust the window. Reactive compaction: - -1. **Trigger**: Context usage exceeds threshold (e.g., 80%) -2. **Summarize**: Older turns (first 50% by token count) -3. **Preserve**: Recent context (last 20% of turns) -4. **Label**: Mark snapshot as "compacted at turn N" - -```markdown -## Session Summary (Turns 1-15, compacted) - -**Goal**: Implement Q&A feature with citations -**Decisions made**: -- Use streaming response for UX -- Citation format: [doc:chunk] inline references -**Key files created**: -- src/services/QaService.ts -- src/shared/types.ts (extended with QaResult) -``` - -### Isolate Pattern - -Delegated work must not pollute parent context: - -| Pattern | Context Sharing | Best For | -|---|---|---| -| **Coordinator** (zero inheritance) | None — workers start fresh | Complex multi-phase tasks | -| **Fork** (full inheritance) | Full — single-level only | Quick parallel splits | -| **Swarm** (peer-to-peer) | Shared task list | Long-running independent work | - -**Key constraint**: Fork is single-level only — recursive forks multiply context cost exponentially. - -## Gotchas - -1. **Most async work skips "pending" state** — work units register directly as "running" -2. **Context builders are memoized but manually invalidated** — add invalidation or face staleness -3. **Truncation is silent until it fires** — hard caps enforced at read time -4. **Isolation boundary must be enforced at call time** — don't just remove tools from prompt - -## Related Patterns - -- [Memory Persistence](memory-persistence-pattern.md) — How memory layers interact with context -- [Multi-agent Coordination](multi-agent-pattern.md) — Context sharing across agents - -## Template: Context Budget - -```markdown -## Context Budget (Session) - -| Category | Budget | Current | Status | -|----------|--------|---------|--------| -| System prompt | 2,000 | 1,850 | ✓ | -| Instruction files | 3,000 | 2,400 | ✓ | -| Memory index | 1,000 | 600 | ✓ | -| Session history | 10,000 | 4,200 | ✓ | -| Working context | 15,000 | 3,100 | ✓ | -| **Total** | **31,000** | **12,150** | 39% used | - -**Compaction trigger**: 80% (24,800 tokens) -**Next action**: Trigger compaction at 24,800 tokens -``` - -## Evidence - -Context engineering patterns are observed in production agent runtimes where: -- Context budgets are explicit, not implicit -- Progressive disclosure reduces startup latency by 60-80% -- Manual cache invalidation prevents subtle staleness bugs -- Isolation patterns enable reliable multi-agent coordination diff --git a/.agents/skills/harness-creator/references/gotchas.md b/.agents/skills/harness-creator/references/gotchas.md deleted file mode 100644 index c8cd51b78..000000000 --- a/.agents/skills/harness-creator/references/gotchas.md +++ /dev/null @@ -1,209 +0,0 @@ -# Gotchas — Harness Engineering Failure Modes - -Non-obvious principles that will cause bugs if you violate them. - ---- - -## 1. Memory Index Caps Fire Silently - -**Symptom**: Recent memories "disappear" without error. - -**Cause**: Index has hard caps (e.g., 200 lines / 25KB) enforced at read time. Long entries (multi-sentence summaries) hit byte cap while staying under line cap. - -**Fix**: Keep index entries to one-line hooks. Put detail in topic files. - -```markdown -✓ Good: "Use bun, not npm - user preference 2024-01-15" -✗ Bad: "The user prefers bun over npm because it's faster. This was discussed on 2024-01-15 when the user said 'use bun not npm' and I updated the package.json accordingly..." -``` - ---- - -## 2. Priority Ordering is Counterintuitive - -**Symptom**: Global rule silently overridden by local file. - -**Cause**: Local overrides beat project rules, which beat user rules, which beat org rules. If you inject at user level expecting it to dominate, a local override file in project root wins. - -**Fix**: Test with full instruction-file stack present: - -```bash -# Test priority ordering -cat ~/.claude/CLAUDE.md # User level -cat ./CLAUDE.md # Project level -cat ./CLAUDE.local.md # Local override (WINS) -``` - ---- - -## 3. Extraction Timing Creates Race Window - -**Symptom**: Background extractor writes memory, but user starts next turn before extraction completes. - -**Cause**: Extraction fires at end of response. User can send message before extraction finishes. - -**Fix**: Coalesce concurrent extraction requests. Advance cursor only after successful run. Failed extraction means those messages reconsidered next time. - ---- - -## 4. Derivable Content Doesn't Belong in Memory - -**Symptom**: Memory index fills with architecture details that stale quickly. - -**Cause**: Agent saves what's derivable from codebase (architecture, code patterns, version history). - -**Fix**: Exclude derivable content by design. Type taxonomy should forbid saving what's in the repo already. - ---- - -## 5. Concurrent Classification is Per-Call, Not Per-Tool - -**Symptom**: Tool marked "concurrent-safe" causes race conditions. - -**Cause**: Same tool can be safe for some inputs and unsafe for others. Don't assume tool's concurrency behavior is static. - -**Fix**: Classify each call at runtime: - -```typescript -// Don't do this: -toolRegistry.register('shell', { concurrentSafe: false }); - -// Do this: -function isCallConcurrentSafe(call: ToolCall): boolean { - if (call.args.command.startsWith('rm -rf')) return false; - if (call.args.command.startsWith('cat')) return true; - // ...runtime classification -} -``` - ---- - -## 6. Permission Evaluation Has Side Effects - -**Symptom**: Permission check changes behavior on subsequent calls. - -**Cause**: Permission evaluator tracks denials, transforms modes, updates state as side effect. Not a pure lookup function. - -**Fix**: Don't cache permission results across calls. Re-evaluate each call fresh. - ---- - -## 7. Most Async Work Skips "Pending" State - -**Symptom**: UI shows "pending" but work unit never enters that state. - -**Cause**: Work units register directly as "running" in practice. "Pending" exists in state machine but rarely used. - -**Fix**: Don't build UI that assumes every work unit starts pending. - ---- - -## 8. Fork Children Must Not Fork - -**Symptom**: Context cost explodes exponentially. - -**Cause**: Recursive forks multiply context: parent + child1 + child2 + grandchildren... - -**Fix**: Enforce single-level invariant. Keep fork tool in child's pool (for prompt cache sharing) but block at call time. - ---- - -## 9. Context Builders are Memoized but Manually Invalidated - -**Symptom**: Model sees stale data for entire session. - -**Cause**: Context builder cached at startup, but mutation doesn't clear cache. - -**Fix**: Every mutation point must explicitly clear its corresponding cache: - -```typescript -// Example: Cache invalidation at mutation point -async function editFile(path: string, content: string) { - await writeFile(path, content); - context.cache.invalidate(`file:${path}`); // MUST invalidate -} -``` - ---- - -## 10. Hook Trust is All-or-Nothing - -**Symptom**: Entire extension system disabled because one hook untrusted. - -**Cause**: If workspace untrusted, all hooks skip — not just suspicious ones. - -**Fix**: Design hooks with trust gate at dispatch point. Don't attempt per-hook trust evaluation. - ---- - -## 11. Eviction Requires Notification - -**Symptom**: Parent can never read work unit result. - -**Cause**: Work unit evicted before parent notified of completion. Race condition: parent tries to read result that's already GC'd. - -**Fix**: Two-phase eviction: -1. Clean disk output at terminal state (eager) -2. Clean in-memory record after parent notified (lazy) - ---- - -## 12. Skill Listing Budgets Are Tight - -**Symptom**: Skill description truncated, can't trigger properly. - -**Cause**: Skill descriptions concatenated and capped per entry (~150 chars). Front-loaded trigger language gets priority. - -**Fix**: Front-load distinctive trigger language: - -```markdown -✓ Good: "harness-patterns: Memory, permissions, context engineering, multi-agent" -✗ Bad: "A comprehensive skill for understanding and implementing various patterns related to AI agent harnesses and runtime systems..." -``` - ---- - -## 13. Default Tool Permission is "Allow" - -**Symptom**: Tool bypasses expected gate. - -**Cause**: Tools without custom permission logic delegate entirely to rule-based system. Default is "allow" unless configured otherwise. - -**Fix**: Override default for sensitive tools: - -```typescript -registry.register('shell', { - defaultPermission: 'ask', // NOT 'allow' - // ... -}); -``` - ---- - -## 14. Team Memory Requires Auto-Memory Enabled - -**Symptom**: Team-shared memory doesn't work even when configured. - -**Cause**: Team memory builds on same directory/index infrastructure as auto-memory. Disabling auto-memory (via env var or settings) also disables team memory. - -**Fix**: Ensure auto-memory enabled before enabling team memory. Check both feature gate and enablement check. - ---- - -## 15. Orphaned Topic Files Accumulate - -**Symptom**: Disk space fills with `.claude/memory/topics/` files. - -**Cause**: Two-step save (topic file then index). Crash between steps leaves orphaned topic file. - -**Fix**: Periodic sweep deletes topic files not referenced by index. Orphans don't corrupt index but consume disk space. - ---- - -## Related Reading - -- [Memory Persistence Pattern](memory-persistence-pattern.md) — Gotchas #1, #3, #4, #15 -- [Tool Registry Pattern](tool-registry-pattern.md) — Gotchas #5, #6, #13 -- [Multi-agent Pattern](multi-agent-pattern.md) — Gotchas #8, #11 -- [Context Engineering Pattern](context-engineering-pattern.md) — Gotchas #9 -- [Lifecycle Pattern](lifecycle-bootstrap-pattern.md) — Gotchas #10, #14 diff --git a/.agents/skills/harness-creator/references/lifecycle-bootstrap-pattern.md b/.agents/skills/harness-creator/references/lifecycle-bootstrap-pattern.md deleted file mode 100644 index 3fdb127c6..000000000 --- a/.agents/skills/harness-creator/references/lifecycle-bootstrap-pattern.md +++ /dev/null @@ -1,264 +0,0 @@ -# Lifecycle and Bootstrap Pattern - -## Problem - -Agent runtimes need extensibility without compromising safety: - -- **Hooks** — Extend behavior at lifecycle moments (pre/post tool execution, session start/end) -- **Background tasks** — Track long-running work without blocking the main agent -- **Bootstrap** — Structure initialization across multiple entry modes (CLI, server, SDK) - -But uncontrolled extensibility creates: -- Security holes from untrusted hooks -- Resource leaks from tasks that never complete -- Race conditions in initialization - -## Golden Rules - -### Hook Trust is All-or-Nothing - -If the workspace is untrusted, **all hooks skip** — not just suspicious ones. Session-scoped hooks are ephemeral and cleaned on session end. - -```typescript -// Example: Hook dispatch with trust gate -async function dispatchHook( - hookType: HookType, - context: HookContext -): Promise { - - // Trust gate: if workspace untrusted, skip ALL hooks - if (!context.trustBoundary.crossed) { - logger.warn('Untrusted workspace, skipping hooks'); - return []; - } - - // Session-scoped hooks ephemeral — cleanup on session end - const sessionHooks = context.hooks.getByScope('session'); - const projectHooks = context.hooks.getByScope('project'); - - return await Promise.all([ - ...sessionHooks.map(h => h.execute(context)), - ...projectHooks.map(h => h.execute(context)), - ]); -} -``` - -### Long-Running Work: Typed State Machines with Two-Phase Eviction - -Each work unit gets: -1. **Typed, prefixed ID** (e.g., `extractor-001`, `benchmark-002`) -2. **Strict lifecycle** (running → completed | failed | killed) -3. **Disk-backed output** (not just in-memory) - -Eviction is two-phase: -1. **Disk output** cleaned eagerly at terminal state -2. **In-memory records** cleaned lazily after parent notified - -### Bootstrap: Dependency-Ordered, Memoized Stages - -Multiple entry modes (CLI, server, SDK) share the same bootstrap path: - -``` -Stage 1: Create minimal context (no trust required) - ↓ -Stage 2: Load tools (read-only safe) - ↓ -Stage 3: Trust boundary crossed (user grants consent) - ↓ -Stage 4: Load security-sensitive subsystems (telemetry, secret env vars) -``` - -**Critical inflection**: Security-sensitive subsystems must not activate before trust is established. - -## When To Use - -- You need to extend agent behavior without modifying core code -- You need to track long-running background work -- You need structured initialization across multiple entry modes -- You need hooks at lifecycle moments (pre/post tool, session start/end) - -## Tradeoffs - -| Decision | Benefit | Cost | -|---|---|---| -| All-or-nothing hook trust | Simple security boundary | One untrusted hook disables entire extension system | -| Disk-backed task output | Memory constant regardless of concurrent work | I/O latency proportional to work units | -| Dependency-ordered bootstrap | Multiple entry modes share path | Initial startup sequential (can't parallelize stages) | -| Memoized stages | Re-init is fast | Must carefully invalidate memoization on config change | - -## Implementation Patterns - -### Hook Lifecycle - -Six hook types dispatched at defined moments: - -```typescript -interface HookRegistry { - // Session lifecycle - onSessionStart: (context: SessionContext) => Promise; - onSessionEnd: (context: SessionContext) => Promise; - - // Tool execution - preToolExecute: (context: ToolContext) => Promise; - postToolExecute: (context: ToolResult) => Promise; - - // Prompt submission - prePromptSubmit: (context: PromptContext) => Promise; - postPromptSubmit: (context: ResponseContext) => Promise; -} - -// Usage: Register hooks via config -// /update-config hooks.preToolExecute = "scripts/audit-tool-call.js" -``` - -### Long-Running Task Tracking - -```typescript -interface TaskRegistry { - // Typed prefixed IDs - registerWork( - type: 'extraction' | 'benchmark' | 'indexing', - outputType: 'json' | 'text' | 'file' - ): string; // Returns typed ID: `extraction-001` - - // Strict state machine - updateState( - taskId: string, - state: 'running' | 'completed' | 'failed' | 'killed', - output?: any - ): void; - - // Two-phase eviction - evictTask(taskId: string): void; - // 1. Clean disk output (eager, at terminal state) - // 2. Clean in-memory record (lazy, after parent notified) -} -``` - -### Bootstrap Sequence - -```typescript -// Example: Dependency-ordered initialization -class AgentBootstrap { - private stages = new Map(); - private memoizedCallers = new Map(); - - async bootstrap(entryMode: 'cli' | 'server' | 'sdk'): Promise { - - // Stage 1: Minimal context (no trust required) - await this.runStage('minimal-context', async () => { - return { - cwd: process.cwd(), - entryMode, - trustBoundary: { crossed: false }, - }; - }); - - // Stage 2: Load tools (read-only safe) - await this.runStage('load-tools', async (context) => { - context.tools = await this.loadSafeTools(); - return context; - }); - - // Stage 3: Trust boundary (user grants consent) - await this.runStage('trust-boundary', async (context) => { - const consent = await this.requestConsent(); - context.trustBoundary = { crossed: consent }; - return context; - }); - - // Stage 4: Security-sensitive subsystems (requires trust) - if (context.trustBoundary.crossed) { - await this.runStage('load-sensitive', async (context) => { - context.telemetry = await this.loadTelemetry(); - context.secretEnvVars = await this.loadSecrets(); - return context; - }); - } - - return context; - } - - private async runStage( - name: string, - fn: (context: AgentContext) => Promise - ): Promise { - // Memoized: skip if already run - if (this.stages.has(name) && this.stages.get(name).complete) { - return; - } - - // Run stage - const stage = { name, complete: false, running: true }; - this.stages.set(name, stage); - - try { - await fn(this.context); - stage.complete = true; - } finally { - stage.running = false; - } - } -} -``` - -## Gotchas - -1. **Hook trust is all-or-nothing** — One untrusted hook disables entire extension system -2. **Most async work skips "pending" state** — Work units register directly as "running" -3. **Eviction requires notification** — Terminal work unit only GC-eligible after parent notified -4. **Fast-path dispatch** — Memoized callers must handle concurrent calls without re-running stages -5. **Hook types must be disjoint** — Don't create overlapping hook scopes - -## Related Patterns - -- [Tool Registry](tool-registry-pattern.md) — How tools are registered at bootstrap -- [Memory Persistence](memory-persistence-pattern.md) — How memory is loaded at init - -## Template: Bootstrap Checklist - -Before declaring bootstrap complete: - -```markdown -## Bootstrap Verification - -### Stage 1: Minimal Context -- [ ] Working directory confirmed -- [ ] Entry mode determined (cli / server / sdk) -- [ ] Trust boundary NOT crossed (no secrets loaded) - -### Stage 2: Tools Loaded -- [ ] Read-only tools registered (read, search, glob) -- [ ] Write tools NOT yet registered (edit, shell) -- [ ] Tool permissions set to default (ask / deny) - -### Stage 3: Trust Boundary -- [ ] User consent requested (interactive or config flag) -- [ ] Consent recorded in session state -- [ ] Security audit logged - -### Stage 4: Sensitive Subsystems -- [ ] Telemetry initialized (if consent given) -- [ ] Secret env vars loaded (if consent given) -- [ ] Write tools registered (edit, shell, exec) -- [ ] Hook system enabled (if workspace trusted) - -### Stage 5: Background Tasks -- [ ] Task registry initialized -- [ ] Cleanup handlers registered -- [ ] Drain-on-shutdown configured - -## If Any Stage Fails - -- Bootstrap halts immediately -- Session remains in safe mode (read-only) -- Error logged with stage name and failure reason -``` - -## Evidence - -Lifecycle and bootstrap patterns are observed in production runtimes where: -- Hook dispatch is all-or-nothing based on workspace trust -- Long-running tasks use typed prefixed IDs and disk-backed output -- Bootstrap is dependency-ordered with memoized stages -- Trust boundary is explicit inflection point for security-sensitive subsystems diff --git a/.agents/skills/harness-creator/references/memory-persistence-pattern.md b/.agents/skills/harness-creator/references/memory-persistence-pattern.md deleted file mode 100644 index 089e0ac7e..000000000 --- a/.agents/skills/harness-creator/references/memory-persistence-pattern.md +++ /dev/null @@ -1,110 +0,0 @@ -# Memory and Persistence Pattern - -## Problem - -Without persistent memory, an agent loses all user preferences, project context, and behavioral feedback the moment a session ends. Users must repeat corrections every session ("use bun, not npm"), and the agent cannot accumulate the working knowledge that makes it genuinely useful over time. - -## Golden Rules - -### Separate layers by scope and durability - -- **Instruction memory** (human-curated, version-controlled): AGENTS.md, CLAUDE.md, project conventions -- **Auto-memory** (agent-written, persistent): Progress logs, session handoffs, discovered patterns -- **Session extraction** (background-derived): Automatic transcript analysis at session end - -### Two-step save invariant - -Every memory write is a two-step operation: -1. Write the full content to a dedicated topic file -2. Append a one-line pointer to the index - -If the process crashes between steps, the worst outcome is an orphaned topic file — the index remains consistent. - -### Local overrides win — always - -When the same topic is addressed at multiple scopes, the most-local instruction takes priority: - -``` -Organization-wide → User-level → Project-level → Local override - ↓ ↓ ↓ ↓ - sets floor narrows it narrows further final say -``` - -### The index is bounded always-on context; topic files are on-demand detail - -- **Index**: Hard-capped at ~200 lines / 25KB, one line per entry -- **Topic files**: Unlimited detail, loaded on demand - -## When To Use - -- Your agent persists across sessions and must recall user preferences or project context -- Multiple scopes of instruction coexist and need clear priority ordering -- The agent should learn from sessions without manual curation -- You need background extraction that doesn't block the user - -## Tradeoffs - -| Decision | Benefit | Cost | -|---|---|---| -| Layered memory | Each scope can be shared, audited, overridden independently | More files to discover at startup | -| Local-wins priority | Users can override without touching shared files | Global rule can be silently overridden | -| Bounded index with on-demand topics | Constant context cost regardless of memory volume | Agent must perform extra retrieval step | -| Background extraction | No latency added to user responses | Race window between extraction and next turn | - -## Implementation Patterns - -1. **Define memory directory** idempotently at startup (e.g., `.claude/memory/`) -2. **Create index file** with hard caps enforced at read time -3. **Implement two-step save**: topic file first, then index update -4. **Fire background extraction** only after final response with no pending tool calls -5. **Enforce mutual exclusion**: if main agent wrote to memory, skip extraction that turn -6. **Build review mechanism** for cross-layer promotion proposals - -## Gotchas - -1. **Index truncation is silent until it fires** — keep entries short -2. **Priority ordering is counterintuitive** — local beats project beats user beats org -3. **Extraction timing creates a race window** — user can start next turn before extraction completes -4. **Derivable content doesn't belong in memory** — architecture and code patterns are re-derivable from the codebase -5. **Orphaned topic files accumulate** — periodic cleanup recommended - -## Related Patterns - -- [Context Engineering](context-engineering-pattern.md) — How to manage context budget across layers -- [Lifecycle & Bootstrap](lifecycle-bootstrap-pattern.md) — How initialization loads memory - -## Template: Progress Log Structure - -```markdown -# Session Progress Log - -## Current State (Last Updated: YYYY-MM-DD HH:MM) - -**Active Feature:** feat-003 - Q&A with Citations -**Status:** In Progress (60% complete) - -### What's Done -- [x] Document chunking pipeline -- [x] Index data structure -- [ ] Q&A handler (in progress) - -### What's In Progress -- Implementing Q&A IPC handler -- Need to decide: streaming vs batch response - -### Blockers -- Waiting on decision: citation format (footnotes vs inline) - -### Next Session Should -1. Complete Q&A handler -2. Add citation formatting -3. Test end-to-end flow -``` - -## Evidence - -This pattern is grounded in production agent runtimes including Claude Code's memory system, which implements: -- Four-level instruction hierarchy (org/user/project/local) -- Four-type auto-memory taxonomy (user/feedback/project/reference) -- Background session extraction with mutual exclusion -- Team-shared memory as an extension layer diff --git a/.agents/skills/harness-creator/references/multi-agent-pattern.md b/.agents/skills/harness-creator/references/multi-agent-pattern.md deleted file mode 100644 index 36a0aa3f8..000000000 --- a/.agents/skills/harness-creator/references/multi-agent-pattern.md +++ /dev/null @@ -1,193 +0,0 @@ -# Multi-Agent Coordination Pattern - -## Problem - -Single agents hit limits: -- **Context limits** — Can't hold full research + implementation in one session -- **Specialization** — Need separate researchers, implementers, reviewers -- **Parallelism** — Want to explore multiple approaches simultaneously - -But multi-agent systems introduce chaos: -- Workers duplicate each other's research -- Coordinators delegate understanding instead of synthesizing -- Context inheritance explodes exponentially - -## Golden Rules - -### The Coordinator Must Synthesize, Not Delegate Understanding - -**Anti-pattern:** -> "Based on your findings, fix the authentication system." - -**Pattern:** -> "Research identified 3 auth flows: login, logout, token refresh. Implement ONLY the token refresh handler using the JWT strategy documented in [research output]. Return: implementation diff + test results." - -The coordinator (orchestrator) adds value by digesting worker results into precise specs before dispatching implementation. - -### Three Delegation Patterns - -| Pattern | Context Sharing | Best For | Constraints | -|---------|----------------|----------|-------------| -| **Coordinator** | None — workers start fresh | Complex multi-phase tasks (research → synthesize → implement → verify) | Slowest but safest | -| **Fork** | Full — child inherits parent history | Quick parallel splits sharing loaded context | **Single-level only** — recursive forks multiply context cost | -| **Swarm** | Peer-to-peer via shared task list | Long-running independent workstreams | **Flat roster** — teammates can't spawn other teammates | - -### Results Arrive Asynchronously; Fire-and-Forget Registration Returns ID Immediately - -```typescript -// Example: Spawn worker, get ID back immediately -const taskId = await coordinator.spawn({ - type: 'research', - prompt: 'Analyze auth flows...', - toolFilter: ['read', 'search'], // Restrict tools -}); - -// Parent can continue working while worker runs -// Results arrive via callback or polling -``` - -## When To Use - -- Task too large for single agent session -- Need parallel exploration (e.g., prototype multiple approaches) -- Want persistent specialized teammates (researcher, implementer, reviewer) -- Complex multi-phase workflows - -## Tradeoffs - -| Pattern | Speed | Safety | Context Cost | -|---------|-------|--------|--------------| -| **Coordinator** | Slowest | Safest | Lowest (zero inheritance) | -| **Fork** | Fastest | Medium | Highest (full inheritance) | -| **Swarm** | Medium | Medium | Medium (shared state only) | - -## Implementation Patterns - -### Coordinator Pattern (Recommended for Complex Tasks) - -Phased workflow: - -``` -Phase 1: Research - ↓ (synthesize findings) -Phase 2: Plan - ↓ (precise specs) -Phase 3: Implement - ↓ (verify) -Phase 4: Review -``` - -```typescript -// Example: Coordinator workflow -const research = await coordinator.spawn({ - role: 'researcher', - prompt: `Analyze existing authentication in ${authDir}. - Find: login flow, logout flow, token handling. - Return: structured findings only. NO implementation suggestions.`, - toolFilter: ['read', 'search', 'glob'], // Can't write -}); - -await coordinator.synthesize(research.results); - -const implement = await coordinator.spawn({ - role: 'implementer', - prompt: `Implement token refresh handler using the JWT strategy - from [Phase 2 findings]. - Constraints: Use existing AuthService patterns, add tests.`, - toolFilter: ['read', 'search', 'edit', 'test'], // Can write -}); -``` - -### Fork Pattern (Single-Level Only) - -```typescript -// Parent spawns children for parallel work -const forks = await Promise.all([ - coordinator.fork({ - prompt: 'Implement login handler', - inheritContext: true, // Full parent history - }), - coordinator.fork({ - prompt: 'Implement logout handler', - inheritContext: true, - }), -]); - -// CRITICAL: Children must not fork recursively -// If allowed, context cost multiplies: parent + child1 + child2 + ... -``` - -### Swarm Pattern (Flat Roster) - -```typescript -// Swarm: persistent team with shared task list -const swarm = new Swarm([ - { id: 'researcher', specialty: 'research' }, - { id: 'implementer', specialty: 'implementation' }, - { id: 'reviewer', specialty: 'verification' }, -]); - -// Agents pick tasks from shared queue -// Results posted back to shared state -await swarm.dispatch({ - taskId: 'feat-001', - pickedBy: 'implementer', -}); -``` - -## Gotchas - -1. **Fork children must not fork** — Recursive guard preserves single-level invariant. Keep fork tool in child's pool (for prompt cache sharing) but block at call time. -2. **Coordinator workers start with zero context** — Only explicit prompt is passed. Don't assume child sees parent's accumulated research. -3. **Swarm teammates cannot spawn other teammates** — Roster is flat to prevent uncontrolled growth. -4. **Write self-contained prompts** — "Based on your findings" is an anti-pattern. Coordinator must digest first. -5. **Filter each worker's tool set** — Researcher doesn't need write; implementer doesn't need broad search. - -## Related Patterns - -- [Context Engineering](context-engineering-pattern.md) — Isolation patterns for delegation -- [Lifecycle & Bootstrap](lifecycle-bootstrap-pattern.md) — How agents are spawned at init - -## Template: Worker Prompt Structure - -```markdown -# Self-Contained Worker Prompt - -## Context (Copied from Coordinator Synthesis) - -**Task**: Implement token refresh handler -**Background**: Research identified JWT-based auth with 24h access tokens. -**Decision**: Use refresh token rotation (new refresh token on each refresh). - -## Your Role - -You are an **implementer**. Your job is to write production code following the specs above. - -## Constraints - -- Use existing patterns from `${authServicePath}` -- Add tests for success and failure cases -- Do NOT modify login/logout handlers (separate task) - -## Your Tools - -- read, search, edit, test -- Shell: npm test, npm run check only - -## Deliverable - -Return: -1. Implementation diff (files changed) -2. Test results (pass/fail) -3. Any blockers or clarifications needed - -**Do NOT return**: Research findings, architectural debates, alternative designs. -``` - -## Evidence - -Multi-agent coordination patterns are observed in production systems where: -- Coordinator workers start with zero context inheritance -- Fork is restricted to single-level to control context explosion -- Swarm agents communicate through shared task lists, not direct prompts -- Results arrive asynchronously with fire-and-forget registration diff --git a/.agents/skills/harness-creator/references/skill-runtime-pattern.md b/.agents/skills/harness-creator/references/skill-runtime-pattern.md deleted file mode 100644 index 963d98ec6..000000000 --- a/.agents/skills/harness-creator/references/skill-runtime-pattern.md +++ /dev/null @@ -1,43 +0,0 @@ -# Skill Runtime Pattern - -Use this pattern when you want to package reusable agent behavior as a skill instead of repeating long instructions in every repository. - -## What Belongs in a Skill - -- Reusable workflows that apply across projects. -- Domain-specific decision procedures. -- Templates, checklists, and reference material the agent should load on demand. -- Small helper scripts when they are stable and safe to run. - -## What Does Not Belong in a Skill - -- Project-specific architecture facts that should live in the target repository. -- Secrets, tokens, private URLs, or user-specific credentials. -- Large manuals that the agent must always read before acting. -- Commands with destructive side effects unless they are clearly documented and require explicit user approval. - -## Runtime Shape - -A production skill should use progressive disclosure: - -1. `SKILL.md` frontmatter explains when the skill should trigger. -2. The body gives the shortest reliable workflow. -3. `references/` contains deeper material loaded only when relevant. -4. `templates/` contains copyable artifacts. -5. `evals/` captures representative quality checks. - -## Design Rules - -- Keep the entry file concise enough to scan quickly. -- Prefer concrete checklists over abstract advice. -- Link every referenced bundled file and verify it exists. -- Make installation instructions explicit about the repository, skill name, and target agent. -- Treat scripts as optional helpers, not hidden behavior. - -## Validation Checklist - -- [ ] `SKILL.md` exists and has valid frontmatter. -- [ ] Every referenced file exists inside the skill directory. -- [ ] Templates are safe to copy into a target repository. -- [ ] Installation command has been tested with `skills add --list` or equivalent. -- [ ] The skill does not depend on private local paths. diff --git a/.agents/skills/harness-creator/references/tool-registry-pattern.md b/.agents/skills/harness-creator/references/tool-registry-pattern.md deleted file mode 100644 index 1a38db9cd..000000000 --- a/.agents/skills/harness-creator/references/tool-registry-pattern.md +++ /dev/null @@ -1,200 +0,0 @@ -# Tool Registry and Safety Pattern - -## Problem - -Agents need tools (shell, file edit, search, etc.) to be productive. But unbounded tool access creates risks: - -- Destructive operations (rm -rf, DROP TABLE, etc.) -- Race conditions from concurrent tool calls -- Silent policy violations from misconfigured permissions - -The solution is a **fail-closed registry** with explicit concurrency classification and a multi-source permission pipeline. - -## Golden Rules - -### Default to Fail-Closed - -Tools are **non-concurrent** and **non-read-only** unless explicitly marked safe. This prevents: -- Accidental parallel execution of state-mutating operations -- Silent data corruption from concurrent writes - -### Concurrency is Per-Call, Not Per-Tool - -The same tool can be safe for some inputs and unsafe for others: - -``` -✓ Safe (can run in parallel): - - cat file1.txt - - grep "pattern" src/ - - ls -la - -✗ Unsafe (must run serially): - - rm -rf build/ - - npm install (network, filesystem mutation) - - sed -i 's/old/new/g' *.ts -``` - -The runtime partitions a batch of tool calls into consecutive groups: safe calls run in parallel; any unsafe call starts a serial segment. - -### Permission Pipeline has Side Effects - -The permission evaluator is **stateful** — it: -- Tracks denials (for audit and rate limiting) -- Transforms modes (e.g., auto → ask after denial) -- Updates session state as a side effect - -**Strict priority order:** -``` -Policy (org-wide) → User settings → Project rules → Local overrides → Session grants -``` - -## When To Use - -- Your agent runtime needs tool registration -- You need concurrency control for parallel tool calls -- You need permission gating (auto-approve, ask-first, deny) -- You need to track tool usage for audit - -## Tradeoffs - -| Decision | Benefit | Cost | -|---|---|---| -| Fail-closed defaults | New tools are safe out of the box | Developers must actively opt into concurrency | -| Per-call classification | Fine-grained control over parallelism | Requires analyzing each call, not just tool registration | -| Multi-source permission layering | Flexible policy composition | Hard to debug when rules conflict | -| Stateful evaluator | Can adapt behavior based on history | Not a pure function — harder to test | - -## Implementation Patterns - -### Tool Registration - -```typescript -// Example: Tool registry entry -interface ToolDefinition { - name: string; - description: string; - handler: (args: any) => Promise; - - // Safety classification - isReadOnly: boolean; // Default: false - isConcurrentSafe: boolean; // Default: false - - // Optional custom permission logic - permissionCheck?: (args: any, context: ToolContext) => PermissionResult; -} - -// Register tools -registry.register('read_file', { - name: 'read_file', - description: 'Read contents of a file', - handler: readFile, - isReadOnly: true, - isConcurrentSafe: true, // Safe to read multiple files in parallel -}); - -registry.register('write_file', { - name: 'write_file', - description: 'Write or overwrite a file', - handler: writeFile, - isReadOnly: false, - isConcurrentSafe: false, // Must run serially to prevent race conditions -}); -``` - -### Permission Pipeline - -```typescript -// Permission evaluation order -async function evaluatePermission( - toolCall: ToolCall, - context: PermissionContext -): Promise { - - // 1. Policy rules (highest priority, org-wide) - const policyResult = await policyEngine.check(toolCall, context); - if (policyResult !== 'defer') return policyResult; - - // 2. User settings - const userResult = await userSettings.check(toolCall, context); - if (userResult !== 'defer') return userResult; - - // 3. Project rules - const projectResult = await projectRules.check(toolCall, context); - if (projectResult !== 'defer') return projectResult; - - // 4. Local overrides - const localResult = await localOverrides.check(toolCall, context); - if (localResult !== 'defer') return localResult; - - // 5. Session grants (lowest priority) - return sessionGrants.check(toolCall, context); -} -``` - -### Bypass-Immune Rules - -Certain paths or operations should never be auto-approved: - -```yaml -# Protected paths (never auto-approve) -protected_paths: - - /etc/** - - /usr/** - - node_modules/** - - .git/** - -# Protected commands (always ask) -protected_commands: - - "rm -rf*" - - "DROP TABLE*" - - "DELETE FROM*" - - "mkfs*" -``` - -## Gotchas - -1. **Most async work skips "pending" state** — work units register directly as "running" -2. **Permission evaluation has side effects** — don't cache results across calls -3. **Concurrency classification requires analyzing inputs**, not just tool name -4. **The default permission for tools is "allow"** — tools without custom logic delegate to rule-based system -5. **Eviction requires notification** — terminal work units only GC-eligible after parent notified - -## Related Patterns - -- [Lifecycle & Bootstrap](lifecycle-bootstrap-pattern.md) — How tools are registered at init -- Hook Lifecycle (`hook-lifecycle-pattern.md`) — Pre/post tool execution hooks - -## Template: Tool Safety Checklist - -Before enabling a new tool: - -```markdown -## Tool Safety Review - -**Tool name**: [e.g., execute_shell] - -### Classification -- [ ] Determined if read-only (true / false / depends on args) -- [ ] Determined if concurrent-safe (true / false / depends on args) -- [ ] Documented unsafe input patterns - -### Permission Requirements -- [ ] Default mode set to "ask" or "deny" -- [ ] Bypass-immune paths/commands defined -- [ ] Custom permission logic implemented (if needed) -- [ ] Audit logging enabled - -### Testing -- [ ] Tested with safe inputs (should auto-approve) -- [ ] Tested with unsafe inputs (should ask/deny) -- [ ] Tested concurrent execution (should serialize if unsafe) -- [ ] Tested error handling (failures logged, state consistent) -``` - -## Evidence - -Tool registry and safety patterns are observed in production agent runtimes including: -- Claude Code's tool registry with explicit concurrency flags -- Multi-source permission evaluation (settings → project → session) -- Protected path/command lists that bypass auto-approve modes -- Per-call concurrency classification that partitions tool batches diff --git a/.agents/skills/harness-creator/scripts/create-harness.mjs b/.agents/skills/harness-creator/scripts/create-harness.mjs deleted file mode 100644 index 1a33a7f64..000000000 --- a/.agents/skills/harness-creator/scripts/create-harness.mjs +++ /dev/null @@ -1,75 +0,0 @@ -#!/usr/bin/env node -import { chmod, mkdir } from 'node:fs/promises'; -import path from 'node:path'; -import { - copyTemplate, - detectPackageManager, - detectProject, - exists, - initScriptFromCommands, - parseArgs, - verificationCommands, - writeText -} from './lib/harness-utils.mjs'; - -const args = parseArgs(process.argv.slice(2)); - -if (args.help) { - console.log(`Usage: node scripts/create-harness.mjs [--target DIR] [--agent-file AGENTS.md|CLAUDE.md] [--package-manager npm|pnpm|yarn|bun] [--force] - -Creates a minimal production harness: - AGENTS.md or CLAUDE.md - feature_list.json - progress.md - session-handoff.md - init.sh - -Existing files are skipped unless --force is set.`); - process.exit(0); -} - -const target = path.resolve(args.target || args._[0] || process.cwd()); -const agentFile = args.agentFile || 'AGENTS.md'; -const force = Boolean(args.force); -const project = await detectProject(target); -project.packageManager = detectPackageManager(target, args.packageManager); -const commands = args.commands - ? String(args.commands).split(',').map((command) => command.trim()).filter(Boolean) - : verificationCommands(project, args.packageManager); - -await mkdir(target, { recursive: true }); - -const replacements = { - AGENT_FILE_NAME: agentFile, - PROJECT_PURPOSE: project.stack === 'generic' - ? 'Project harness for reliable agent-assisted development.' - : `Project harness for reliable agent-assisted development in a ${project.stack} codebase.`, - VERIFICATION_COMMANDS: commands.map((command) => `- \`${command}\``).join('\n'), - PRIMARY_VERIFICATION_COMMAND: './init.sh' -}; - -const results = []; -results.push(await copyTemplate('agents.md', path.join(target, agentFile), replacements, { force })); -results.push(await copyTemplate('feature-list.json', path.join(target, 'feature_list.json'), {}, { force })); -results.push(await copyTemplate('progress.md', path.join(target, 'progress.md'), {}, { force })); -results.push(await copyTemplate('session-handoff.md', path.join(target, 'session-handoff.md'), {}, { force })); - -const initPath = path.join(target, 'init.sh'); -if (force || !await exists(initPath)) { - await writeText(initPath, initScriptFromCommands(commands)); - await chmod(initPath, 0o755); - results.push({ path: initPath, status: 'written' }); -} else { - results.push({ path: initPath, status: 'skipped', reason: 'exists' }); -} - -console.log(`Created harness for ${target}`); -console.log(`Detected stack: ${project.stack}`); -console.log(`Verification commands:`); -for (const command of commands) { - console.log(` - ${command}`); -} -console.log(''); -for (const result of results) { - console.log(`${result.status.toUpperCase()} ${path.relative(target, result.path)}${result.reason ? ` (${result.reason})` : ''}`); -} diff --git a/.agents/skills/harness-creator/scripts/render-assessment-html.mjs b/.agents/skills/harness-creator/scripts/render-assessment-html.mjs deleted file mode 100644 index 4d8d85fdb..000000000 --- a/.agents/skills/harness-creator/scripts/render-assessment-html.mjs +++ /dev/null @@ -1,26 +0,0 @@ -#!/usr/bin/env node -import path from 'node:path'; -import { - htmlReport, - loadHarnessFiles, - parseArgs, - scoreHarness, - writeText -} from './lib/harness-utils.mjs'; - -const args = parseArgs(process.argv.slice(2)); - -if (args.help) { - console.log(`Usage: node scripts/render-assessment-html.mjs [--target DIR] [--output FILE] - -Renders the five-subsystem harness assessment as a standalone HTML file.`); - process.exit(0); -} - -const target = path.resolve(args.target || args._[0] || process.cwd()); -const output = path.resolve(args.output || path.join(target, 'harness-assessment.html')); -const result = scoreHarness(await loadHarnessFiles(target)); - -await writeText(output, htmlReport(result, `Harness Assessment: ${path.basename(target)}`)); -console.log(`HTML report written to ${output}`); -console.log(`Overall: ${result.overall}/100`); diff --git a/.agents/skills/harness-creator/scripts/run-benchmark.mjs b/.agents/skills/harness-creator/scripts/run-benchmark.mjs deleted file mode 100644 index 2e6ab0f6c..000000000 --- a/.agents/skills/harness-creator/scripts/run-benchmark.mjs +++ /dev/null @@ -1,121 +0,0 @@ -#!/usr/bin/env node -import path from 'node:path'; -import { fileURLToPath } from 'node:url'; -import { - formatScoreReport, - htmlReport, - loadHarnessFiles, - parseArgs, - readJson, - scoreHarness, - writeText -} from './lib/harness-utils.mjs'; - -const args = parseArgs(process.argv.slice(2)); -const scriptDir = path.dirname(fileURLToPath(import.meta.url)); -const skillRoot = path.resolve(scriptDir, '..'); - -if (args.help) { - console.log(`Usage: node scripts/run-benchmark.mjs [--target DIR] [--output FILE] [--html FILE] - -Runs a lightweight harness benchmark: - 1. Scores the current target harness. - 2. Checks eval coverage in evals/evals.json. - 3. Produces a JSON report and optional HTML report. - -This is a structural benchmark, not an LLM judge. Use it before/after real agent sessions.`); - process.exit(0); -} - -const target = path.resolve(args.target || args._[0] || process.cwd()); -const output = path.resolve(args.output || path.join(target, 'harness-benchmark.json')); -const evalPath = path.resolve(args.evals || path.join(skillRoot, 'evals', 'evals.json')); - -const harnessResult = scoreHarness(await loadHarnessFiles(target)); -const evals = await readJson(evalPath); -const evalResult = scoreEvals(evals); -const report = { - generatedAt: new Date().toISOString(), - target, - harness: harnessResult, - evals: evalResult, - recommendation: recommend(harnessResult, evalResult) -}; - -await writeText(output, `${JSON.stringify(report, null, 2)}\n`); -console.log(`Benchmark report written to ${output}`); -console.log(''); -console.log(formatScoreReport(harnessResult, target)); -console.log(`Eval coverage: ${evalResult.score}/100 (${evalResult.passed}/${evalResult.total})`); -console.log(`Recommendation: ${report.recommendation}`); - -if (args.html) { - const htmlPath = path.resolve(args.html); - await writeText(htmlPath, renderBenchmarkHtml(report)); - console.log(`HTML benchmark report written to ${htmlPath}`); -} - -if (harnessResult.overall < Number(args.minScore || 70) || evalResult.score < Number(args.minEvalScore || 80)) { - process.exitCode = 1; -} - -function scoreEvals(evalsJson) { - const cases = Array.isArray(evalsJson.evals) ? evalsJson.evals : []; - const checks = []; - checks.push({ pass: cases.length >= 10, message: 'At least 10 eval cases' }); - checks.push({ pass: cases.some((item) => /minimal|creation/i.test(item.name)), message: 'Covers minimal harness creation' }); - checks.push({ pass: cases.some((item) => /session|continuity/i.test(item.name)), message: 'Covers session continuity' }); - checks.push({ pass: cases.some((item) => /assessment|score/i.test(item.name)), message: 'Covers harness assessment' }); - checks.push({ pass: cases.some((item) => /verification/i.test(item.name)), message: 'Covers verification workflow' }); - checks.push({ pass: cases.some((item) => /memory/i.test(item.name)), message: 'Covers memory taxonomy' }); - checks.push({ pass: cases.some((item) => /tool|permission|safety/i.test(item.name)), message: 'Covers tool safety' }); - checks.push({ pass: cases.some((item) => /multi-agent|delegation|coordination/i.test(item.name)), message: 'Covers multi-agent coordination' }); - checks.push({ pass: cases.every((item) => item.prompt && item.expected_output && Array.isArray(item.expectations)), message: 'Each eval has prompt, expected output, expectations' }); - checks.push({ pass: cases.every((item) => item.expectations?.length >= 3), message: 'Each eval has at least three expectation checks' }); - - const passed = checks.filter((check) => check.pass).length; - return { - score: Math.round((passed / checks.length) * 100), - passed, - total: checks.length, - cases: cases.length, - checks - }; -} - -function recommend(harnessResult, evalResult) { - if (harnessResult.overall >= 85 && evalResult.score >= 90) { - return 'Ready for realistic before/after agent-session benchmarking.'; - } - if (harnessResult.overall < 70) { - return `Improve the ${harnessResult.bottleneck} subsystem before benchmarking agent behavior.`; - } - if (evalResult.score < 80) { - return 'Expand eval coverage before treating benchmark results as representative.'; - } - return 'Usable, with some gaps worth tightening after first real sessions.'; -} - -function renderBenchmarkHtml(report) { - const evalHtml = htmlReport(report.harness, `Harness Benchmark: ${path.basename(report.target)}`) - .replace('', `
-

Eval Coverage ${report.evals.score}/100

-

${report.evals.passed}/${report.evals.total} benchmark checks passed across ${report.evals.cases} eval cases.

-
    ${report.evals.checks.map((check) => `
  • ${check.pass ? 'PASS' : 'FAIL'} ${escapeHtml(check.message)}
    • `).join('')}
-
-
-

Recommendation

-

${escapeHtml(report.recommendation)}

-
- `); - return evalHtml; -} - -function escapeHtml(value) { - return String(value) - .replaceAll('&', '&') - .replaceAll('<', '<') - .replaceAll('>', '>') - .replaceAll('"', '"') - .replaceAll("'", '''); -} diff --git a/.agents/skills/harness-creator/scripts/validate-harness.mjs b/.agents/skills/harness-creator/scripts/validate-harness.mjs deleted file mode 100644 index 3ae16d30f..000000000 --- a/.agents/skills/harness-creator/scripts/validate-harness.mjs +++ /dev/null @@ -1,43 +0,0 @@ -#!/usr/bin/env node -import path from 'node:path'; -import { - formatScoreReport, - htmlReport, - loadHarnessFiles, - parseArgs, - scoreHarness, - writeText -} from './lib/harness-utils.mjs'; - -const args = parseArgs(process.argv.slice(2)); - -if (args.help) { - console.log(`Usage: node scripts/validate-harness.mjs [--target DIR] [--json] [--html FILE] - -Scores a project harness across five subsystems: - instructions, state, verification, scope, lifecycle - -Exit code is 0 when the harness scores at least --min-score (default 70).`); - process.exit(0); -} - -const target = path.resolve(args.target || args._[0] || process.cwd()); -const minScore = Number(args.minScore || 70); -const files = await loadHarnessFiles(target); -const result = scoreHarness(files); - -if (args.html) { - const htmlPath = path.resolve(args.html); - await writeText(htmlPath, htmlReport(result, `Harness Assessment: ${path.basename(target)}`)); - console.log(`HTML report written to ${htmlPath}`); -} - -if (args.json) { - console.log(JSON.stringify(result, null, 2)); -} else { - console.log(formatScoreReport(result, target)); -} - -if (result.overall < minScore) { - process.exitCode = 1; -} diff --git a/.agents/skills/harness-creator/templates/agents.md b/.agents/skills/harness-creator/templates/agents.md deleted file mode 100644 index 0b41e940b..000000000 --- a/.agents/skills/harness-creator/templates/agents.md +++ /dev/null @@ -1,68 +0,0 @@ -# {{AGENT_FILE_NAME}} - -{{PROJECT_PURPOSE}} - -## Startup Workflow - -Before writing code: - -1. **Confirm working directory** with `pwd` -2. **Read this file** completely -3. **Read project docs if present** (`docs/ARCHITECTURE.md`, `docs/PRODUCT.md`, README, or equivalent) -4. **Run `./init.sh`** to verify environment is healthy -5. **Read `feature_list.json`** to see current feature state -6. **Review recent commits** with `git log --oneline -5` - -If baseline verification is failing, repair that first before adding new scope. - -## Working Rules - -- **One feature at a time**: Pick exactly one unfinished feature from `feature_list.json` -- **Verification required**: Don't claim done without running verification commands -- **Update artifacts**: Before ending session, update `progress.md` and `feature_list.json` -- **Stay in scope**: Don't modify files unrelated to the current feature -- **Leave clean state**: Next session must be able to run `./init.sh` immediately - -## Required Artifacts - -- `feature_list.json` — Feature state tracker (source of truth) -- `progress.md` — Session continuity log -- `init.sh` — Standard startup and verification path -- `session-handoff.md` — Optional, for larger sessions - -## Definition of Done - -A feature is done only when ALL of the following are true: - -- [ ] Target behavior is implemented -- [ ] Required verification actually ran (tests / lint / type-check) -- [ ] Evidence recorded in `feature_list.json` or `progress.md` -- [ ] Repository remains restartable from standard startup path - -## End of Session - -Before ending a session: - -1. Update `progress.md` with current state -2. Update `feature_list.json` with new feature status -3. Record any unresolved risks or blockers -4. Commit with descriptive message once work is in safe state -5. Leave repo clean enough for next session to run `./init.sh` immediately - -## Verification Commands - -```bash -# Full verification (recommended) -{{PRIMARY_VERIFICATION_COMMAND}} -``` - -Required checks: -{{VERIFICATION_COMMANDS}} - -## Escalation - -If you encounter: -- **Architecture decisions**: Consult project architecture docs if present, otherwise ask user -- **Unclear requirements**: Check product/requirements docs if present, otherwise ask user -- **Repeated test failures**: Update progress, flag for human review -- **Scope ambiguity**: Re-read `feature_list.json` for definition of done diff --git a/.agents/skills/harness-creator/templates/feature-list.json b/.agents/skills/harness-creator/templates/feature-list.json deleted file mode 100644 index fff05b01c..000000000 --- a/.agents/skills/harness-creator/templates/feature-list.json +++ /dev/null @@ -1,44 +0,0 @@ -{ - "features": [ - { - "id": "feat-001", - "name": "Project Setup", - "description": "Confirm the project can install dependencies, run verification, and start from a clean checkout", - "dependencies": [], - "status": "not-started", - "evidence": "" - }, - { - "id": "feat-002", - "name": "First User-Facing Feature", - "description": "Replace this placeholder with the first concrete behavior the agent should implement", - "dependencies": ["feat-001"], - "status": "not-started", - "evidence": "" - }, - { - "id": "feat-003", - "name": "Verification Coverage", - "description": "Add or confirm tests, type checks, linting, or manual verification for the active feature", - "dependencies": ["feat-002"], - "status": "not-started", - "evidence": "" - }, - { - "id": "feat-004", - "name": "Documentation Update", - "description": "Update README, architecture notes, or product docs affected by the implemented feature", - "dependencies": ["feat-003"], - "status": "not-started", - "evidence": "" - }, - { - "id": "feat-005", - "name": "Cleanup and Handoff", - "description": "Record verification evidence, update progress.md, and leave a clear next-session path", - "dependencies": ["feat-004"], - "status": "not-started", - "evidence": "" - } - ] -} diff --git a/.agents/skills/harness-creator/templates/feature-list.schema.json b/.agents/skills/harness-creator/templates/feature-list.schema.json deleted file mode 100644 index 45b852161..000000000 --- a/.agents/skills/harness-creator/templates/feature-list.schema.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "title": "Feature List", - "description": "Track feature implementation state for agent-driven development", - "type": "object", - "properties": { - "features": { - "type": "array", - "items": { - "type": "object", - "properties": { - "id": { - "type": "string", - "description": "Unique feature identifier (e.g., feat-001)", - "pattern": "^feat-\\d+$" - }, - "name": { - "type": "string", - "description": "Short feature name" - }, - "description": { - "type": "string", - "description": "What this feature does" - }, - "dependencies": { - "type": "array", - "items": { - "type": "string" - }, - "description": "Feature IDs that must be done before this one" - }, - "status": { - "type": "string", - "enum": ["not-started", "in-progress", "blocked", "done"], - "description": "Current implementation state" - }, - "evidence": { - "type": "string", - "description": "Verification evidence when status is 'done'" - } - }, - "required": ["id", "name", "description", "status"] - } - } - }, - "required": ["features"] -} diff --git a/.agents/skills/harness-creator/templates/init.sh b/.agents/skills/harness-creator/templates/init.sh deleted file mode 100644 index a305a37f9..000000000 --- a/.agents/skills/harness-creator/templates/init.sh +++ /dev/null @@ -1,75 +0,0 @@ -#!/bin/bash -set -e - -echo "=== Harness Initialization ===" - -if [ -f package.json ]; then - if [ -f pnpm-lock.yaml ]; then - PM="pnpm" - elif [ -f yarn.lock ]; then - PM="yarn" - elif [ -f bun.lock ] || [ -f bun.lockb ]; then - PM="bun" - else - PM="npm" - fi - - echo "=== Installing dependencies with $PM ===" - if [ "$PM" = "npm" ]; then - npm install - else - "$PM" install - fi - - node -e "const s=require('./package.json').scripts||{}; process.exit(s.check||s.typecheck||s['type-check']?0:1)" && { - if node -e "const s=require('./package.json').scripts||{}; process.exit(s.check?0:1)"; then - [ "$PM" = "npm" ] && npm run check || "$PM" run check - elif node -e "const s=require('./package.json').scripts||{}; process.exit(s.typecheck?0:1)"; then - [ "$PM" = "npm" ] && npm run typecheck || "$PM" run typecheck - else - [ "$PM" = "npm" ] && npm run type-check || "$PM" run type-check - fi - } - - node -e "const s=require('./package.json').scripts||{}; process.exit(s.lint?0:1)" && { - [ "$PM" = "npm" ] && npm run lint || "$PM" run lint - } - - node -e "const s=require('./package.json').scripts||{}; process.exit(s.test?0:1)" && { - [ "$PM" = "npm" ] && npm test || "$PM" test - } - - node -e "const s=require('./package.json').scripts||{}; process.exit(s.build?0:1)" && { - [ "$PM" = "npm" ] && npm run build || "$PM" run build - } -elif [ -f pyproject.toml ] || [ -f requirements.txt ]; then - echo "=== Running Python verification ===" - python -m pytest - python -m compileall . -elif [ -f go.mod ]; then - echo "=== Running Go verification ===" - go test ./... -elif [ -f Cargo.toml ]; then - echo "=== Running Rust verification ===" - cargo test -elif [ -f pom.xml ]; then - echo "=== Running Maven verification ===" - mvn test -elif [ -f build.gradle ] || [ -f build.gradle.kts ]; then - echo "=== Running Gradle verification ===" - ./gradlew test -elif ls *.csproj *.sln >/dev/null 2>&1; then - echo "=== Running .NET verification ===" - dotnet test -else - echo "No recognized package manifest detected." - echo "Replace this section with the project's verification commands." -fi - -echo "=== Verification Complete ===" -echo "" -echo "Next steps:" -echo "1. Read feature_list.json to see current feature state" -echo "2. Pick ONE unfinished feature to work on" -echo "3. Implement only that feature" -echo "4. Re-run verification before claiming done" diff --git a/.agents/skills/harness-creator/templates/progress.md b/.agents/skills/harness-creator/templates/progress.md deleted file mode 100644 index 3710d5c35..000000000 --- a/.agents/skills/harness-creator/templates/progress.md +++ /dev/null @@ -1,51 +0,0 @@ -# Session Progress Log - -## Current State - -**Last Updated:** YYYY-MM-DD HH:MM -**Session ID:** [optional] -**Active Feature:** [feat-XXX - Feature Name] - -## Status - -### What's Done - -- [x] [Completed item 1] -- [x] [Completed item 2] - -### What's In Progress - -- [ ] [Current work item] - - Details: [specific task] - - Blockers: [if any] - -### What's Next - -1. [Next action item] -2. [Following action item] - -## Blockers / Risks - -- [ ] [Blocker 1]: [description, impact] -- [ ] [Risk 1]: [description, mitigation] - -## Decisions Made - -- **[Decision 1]**: [description] - - Context: [why this decision was made] - - Alternatives considered: [what else was discussed] - -## Files Modified This Session - -- `path/to/file1.ts` - [brief description of change] -- `path/to/file2.ts` - [brief description of change] - -## Evidence of Completion - -- [ ] Tests pass: `[command and output]` -- [ ] Type check clean: `[command and output]` -- [ ] Manual verification: `[what was tested]` - -## Notes for Next Session - -[Free-form notes that will help the next session pick up context] diff --git a/.agents/skills/harness-creator/templates/session-handoff.md b/.agents/skills/harness-creator/templates/session-handoff.md deleted file mode 100644 index d9168cb72..000000000 --- a/.agents/skills/harness-creator/templates/session-handoff.md +++ /dev/null @@ -1,40 +0,0 @@ -# Session Handoff - -## Current Objective - -- Goal: -- Current status: -- Branch / commit: - -## Completed This Session - -- [ ] - -## Verification Evidence - -| Check | Command | Result | Notes | -|---|---|---|---| -| | | | | - -## Files Changed - -- - -## Decisions Made - -- - -## Blockers / Risks - -- - -## Next Session Startup - -1. Read `AGENTS.md`. -2. Read `feature_list.json` and `progress.md`. -3. Review this handoff. -4. Run `./init.sh` or the documented verification command before editing. - -## Recommended Next Step - -- diff --git a/AGENTS.md b/AGENTS.md index 3fcc88e89..e1dc2115e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,60 +1,81 @@ -# Repository Guidelines +# Open Deep Research 仓库概览 -## Startup Workflow +## 项目描述 -Before writing code: +Open Deep Research 是一个可配置、完全开源的深度研究智能体(deep research agent)，支持多个模型提供商、搜索工具以及 MCP（Model Context Protocol，模型上下文协议）服务器。它能够通过并行处理自动执行研究任务，并生成内容全面的研究报告。 -1. Confirm the repository root and read `README.md`. -2. Read `feature_list.json`, `progress.md`, and any current `session-handoff.md`. -3. Select exactly one feature with an actionable status. -4. Run `./init.sh`. If the baseline fails, record the failure before changing code. -5. Review `git status --short` and preserve unrelated user changes. +## 仓库结构 -If dependencies are missing, create the development environment with `uv sync --extra dev`. +### 根目录 -## Project Boundaries +* `README.md` - 完整的项目文档，包含快速入门指南 +* `pyproject.toml` - Python 项目配置与依赖声明 +* `langgraph.json` - LangGraph 配置文件，用于定义主图(Graph)的入口点 +* `uv.lock` - UV 包管理器的依赖锁定文件 +* `LICENSE` - MIT 许可证 +* `.env.example` - 环境变量模板（不受版本控制追踪） -Production code is in `src/open_deep_research/`; deployment authentication is in `src/security/`. Evaluation tooling is in `tests/`, while legacy code and its pytest suite are in `src/legacy/`. `deep_research_from_scratch/` is a separate tutorial package and must not be changed unless the active feature explicitly targets it. +### 核心实现（`src/open_deep_research/`） -Work on one feature at a time. Stay in scope, do not rewrite unrelated files, and never commit `.env`, API keys, private MCP settings, or sensitive generated reports. +* `deep_researcher.py` - LangGraph 的主要实现文件（入口点：`deep_researcher`） +* `configuration.py` - 配置管理与设置 +* `state.py` - 图状态(Graph state)定义与数据结构 +* `prompts.py` - 系统提示词与提示词模板 +* `utils.py` - 工具函数与辅助功能 +* `files/` - 研究输出文件与示例文件 -## State Artifacts +### 遗留实现（`src/legacy/`） -- `feature_list.json` is the source of truth for feature status, dependencies, and evidence. -- `progress.md` records current state, decisions, blockers, and the next action. -- `session-handoff.md` provides a restart path for unfinished work. -- `init.sh` is the standard verification entrypoint. +包含两种早期的研究实现： -Add a concrete feature entry before starting new implementation work. Allowed status values are `not-started`, `in-progress`, `blocked`, and `completed`. +* `graph.py` - 带有人机协同(human-in-the-loop)机制的规划与执行(plan-and-execute)工作流 +* `multi_agent.py` - 监督者—研究员(supervisor-researcher)多智能体架构 +* `legacy.md` - 遗留实现的说明文档 +* `CLAUDE.md` - 面向遗留实现的 Claude 专用指令 +* `tests/` - 遗留实现专用测试 -## Verification Commands +### 安全模块（`src/security/`） -Run the full local baseline with: +* `auth.py` - 用于 LangGraph 部署的身份验证处理程序 -```bash -./init.sh -``` +### 测试（`tests/`） -It performs: +* `run_evaluate.py` - 主评估脚本，配置为在 Deep Research Bench 上运行 +* `evaluators.py` - 专用评估函数 +* `prompts.py` - 评估提示词与评估标准 +* `pairwise_evaluation.py` - 对比评估工具 +* `supervisor_parallel_evaluation.py` - 多线程并行评估 -- `python -m compileall -q src` -- `python -m ruff check .` -- `python -m mypy src` -- `python -m pytest --collect-only -q src/legacy/tests` +### 示例（`examples/`） -The comprehensive evaluation, `python tests/run_evaluate.py`, uses external services and may incur cost. Run it only when explicitly required and credentials are available. +* `arxiv.md` - ArXiv 研究示例 +* `pubmed.md` - PubMed 研究示例 +* `inference-market.md` - 推理市场分析示例 -## Definition of Done +### 手把手教学（deep_research_from_scratch/） +* 改文件夹下放的该项目进面向人类初学者的教学jupyter notebook内容，可视为与本项目的实际运行和使用完全解耦的模块 -A feature is done only when its scoped behavior is complete, relevant verification has run, evidence is recorded, documentation is updated when necessary, and the repository has a clear restart path. +## 核心技术 -## End of Session +* **LangGraph** - 工作流编排与图执行(Graph execution) +* **LangChain** - 大语言模型集成与工具调用(tool calling) +* **多个 LLM 提供商** - 支持 OpenAI、Anthropic、Google、Groq 和 DeepSeek +* **搜索 API** - 支持 Tavily、OpenAI/Anthropic 原生搜索、DuckDuckGo 和 Exa +* **MCP Servers** - 通过 Model Context Protocol 扩展智能体能力 -Before ending: +## 开发命令 -1. Update the active feature status and evidence. -2. Update `progress.md` with files changed, verification results, blockers, and the recommended next step. -3. Refresh `session-handoff.md` when work remains unfinished. -4. Check `git status --short` and leave unrelated changes untouched. +* `uvx langgraph dev` - 启动带有 LangGraph Studio 的开发服务器 +* `python tests/run_evaluate.py` - 运行完整评估 +* `ruff check` - 执行代码 lint 检查 +* `mypy` - 执行类型检查 -Leave a clean restart path: the next session must recover the active objective and rerun verification from these files without relying on chat history. +## 配置 + +所有设置均可通过以下方式进行配置： + +* 环境变量（`.env` 文件） +* LangGraph Studio 中的 Web UI +* 直接修改配置文件 + +关键设置包括模型选择、搜索 API 选择、并发限制以及 MCP server 配置。 diff --git a/AGENTS.override.md b/AGENTS.override.md deleted file mode 100644 index 8690221c1..000000000 --- a/AGENTS.override.md +++ /dev/null @@ -1,81 +0,0 @@ -# Open Deep Research 仓库概览 - -## 项目描述 - -Open Deep Research 是一个可配置、完全开源的深度研究智能体(deep research agent)，支持多个模型提供商、搜索工具以及 MCP（Model Context Protocol，模型上下文协议）服务器。它能够通过并行处理自动执行研究任务，并生成内容全面的研究报告。 - -## 仓库结构 - -### 根目录 - -* `README.md` - 完整的项目文档，包含快速入门指南 -* `pyproject.toml` - Python 项目配置与依赖声明 -* `langgraph.json` - LangGraph 配置文件，用于定义主图(Graph)的入口点 -* `uv.lock` - UV 包管理器的依赖锁定文件 -* `LICENSE` - MIT 许可证 -* `.env.example` - 环境变量模板（不受版本控制追踪） - -### 核心实现（`src/open_deep_research/`） - -* `deep_researcher.py` - LangGraph 的主要实现文件（入口点：`deep_researcher`） -* `configuration.py` - 配置管理与设置 -* `state.py` - 图状态(Graph state)定义与数据结构 -* `prompts.py` - 系统提示词与提示词模板 -* `utils.py` - 工具函数与辅助功能 -* `files/` - 研究输出文件与示例文件 - -### 遗留实现（`src/legacy/`） - -包含两种早期的研究实现： - -* `graph.py` - 带有人机协同(human-in-the-loop)机制的规划与执行(plan-and-execute)工作流 -* `multi_agent.py` - 监督者—研究员(supervisor-researcher)多智能体架构 -* `legacy.md` - 遗留实现的说明文档 -* `CLAUDE.md` - 面向遗留实现的 Claude 专用指令 -* `tests/` - 遗留实现专用测试 - -### 安全模块（`src/security/`） - -* `auth.py` - 用于 LangGraph 部署的身份验证处理程序 - -### 测试（`tests/`） - -* `run_evaluate.py` - 主评估脚本，配置为在 Deep Research Bench 上运行 -* `evaluators.py` - 专用评估函数 -* `prompts.py` - 评估提示词与评估标准 -* `pairwise_evaluation.py` - 对比评估工具 -* `supervisor_parallel_evaluation.py` - 多线程并行评估 - -### 示例（`examples/`） - -* `arxiv.md` - ArXiv 研究示例 -* `pubmed.md` - PubMed 研究示例 -* `inference-market.md` - 推理市场分析示例 - -### 手把手教学（deep_research_from_scratch/） -* 改文件夹下放的该项目进面向人类初学者的教学jupyter notebook内容，可视为与本项目的实际运行和使用完全解耦的模块 - -## 核心技术 - -* **LangGraph** - 工作流编排与图执行(Graph execution) -* **LangChain** - 大语言模型集成与工具调用(tool calling) -* **多个 LLM 提供商** - 支持 OpenAI、Anthropic、Google、Groq 和 DeepSeek -* **搜索 API** - 支持 Tavily、OpenAI/Anthropic 原生搜索、DuckDuckGo 和 Exa -* **MCP Servers** - 通过 Model Context Protocol 扩展智能体能力 - -## 开发命令 -* `conda activate open-deep-research`启动本项目的专用环境 -* `langgraph dev --allow-blocking `使用langgraph studio UI，启动带有 LangGraph Studio 的开发服务器 -* `python tests/run_evaluate.py` - 运行完整评估 -* `ruff check` - 执行代码 lint 检查 -* `mypy` - 执行类型检查 - -## 配置 - -所有设置均可通过以下方式进行配置： - -* 环境变量（`.env` 文件） -* LangGraph Studio 中的 Web UI -* 直接修改配置文件 - -关键设置包括模型选择、搜索 API 选择、并发限制以及 MCP server 配置。 diff --git a/README.md b/README.md index ce551510f..256e0adbe 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,12 @@ Deep research has broken out as one of the most popular agent applications. This ### 🚀 Quickstart 1. Clone the repository and activate a virtual environment: +```bash +git clone https://github.com/langchain-ai/open_deep_research.git +cd open_deep_research +uv venv +source .venv/bin/activate # On Windows: .venv\Scripts\activate +``` ```bash git clone https://github.com/langchain-ai/open_deep_research.git @@ -32,6 +38,11 @@ python -m pip install --upgrade pip setuptools wheel ``` 2. Install dependencies: +```bash +uv sync +# or +uv pip install -r pyproject.toml +``` ```bash pip install -e . # 可编辑模式安装当前目录中的项目，并基于pyproject.toml进行依赖解析 @@ -45,7 +56,8 @@ cp .env.example .env 4. Launch agent with the LangGraph server locally: ```bash -langgraph dev --allow-blocking +# Install dependencies and start the LangGraph server +uvx --refresh --from "langgraph-cli[inmem]" --with-editable . --python 3.11 langgraph dev --allow-blocking ``` This will open the LangGraph Studio UI in your browser. diff --git a/feature_list.json b/feature_list.json deleted file mode 100644 index 41546282d..000000000 --- a/feature_list.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "features": [ - { - "id": "harness-001", - "name": "Initialize minimal coding-agent harness", - "description": "Add repository instructions, persistent feature state, local verification, and a session handoff path without changing business code.", - "dependencies": [], - "status": "completed", - "evidence": "Five minimal harness artifacts created without business-code changes; harness validator passed 25/25 structural checks (100/100)." - } - ] -} diff --git a/init.sh b/init.sh deleted file mode 100644 index a4d894998..000000000 --- a/init.sh +++ /dev/null @@ -1,19 +0,0 @@ -#!/bin/bash -set -e - -echo "=== Open Deep Research Harness Verification ===" - -echo "=== Compile Python sources ===" -python -m compileall -q src - -echo "=== Ruff ===" -python -m ruff check . - -echo "=== Mypy ===" -python -m mypy src - -echo "=== Collect legacy tests without external API execution ===" -python -m pytest --collect-only -q src/legacy/tests - -echo "=== Verification Complete ===" -echo "Read feature_list.json, select one feature, and record evidence in progress.md." diff --git a/progress.md b/progress.md deleted file mode 100644 index 64316a8b8..000000000 --- a/progress.md +++ /dev/null @@ -1,52 +0,0 @@ -# Session Progress Log - -## Current State - -**Last Updated:** 2026-06-21 -**Active Feature:** `harness-001` — Initialize minimal coding-agent harness -**Status:** Completed - -## What's Done - -- Added the five minimal harness artifacts. -- Restricted routine verification to local, non-billable checks. -- Preserved existing business code and unrelated working-tree changes. - -## Final Validation - -- Passed all 25 structural harness checks. -- Confirmed `feature_list.json` parses successfully and no business code was changed. - -## What's Next - -1. Add a new feature entry before beginning business-code work. -2. Run `uv sync --extra dev` if the development environment is unavailable. -3. Run `./init.sh` and record the result. - -## Blockers / Risks - -- The current shell may not expose project tools on `PATH`; use the environment created by `uv sync --extra dev`. -- Legacy report-quality tests invoke external services, so `init.sh` only collects them. - -## Decisions Made - -- `tests/run_evaluate.py` is excluded from routine startup because it requires credentials and may incur API cost. -- Harness state is stored in versioned files rather than chat history. - -## Files Modified This Session - -- `AGENTS.md` -- `feature_list.json` -- `progress.md` -- `init.sh` -- `session-handoff.md` - -## Verification Evidence - -- Structural validation: `100/100` (`25/25` checks passed) -- Feature tracker JSON parsing: passed -- Business-code tests: not required; no business code changed - -## Notes for Next Session - -Read the state files, add one concrete feature, and run `./init.sh` before implementation. diff --git a/session-handoff.md b/session-handoff.md deleted file mode 100644 index 704a4b189..000000000 --- a/session-handoff.md +++ /dev/null @@ -1,45 +0,0 @@ -# Session Handoff - -## Current Objective - -- Goal: Initialize and validate the repository's minimal coding-agent harness. -- Current status: Completed; all structural checks passed. -- Branch / commit: Use the current working tree; no commit created. - -## Completed This Session - -- Added instructions, feature state, progress tracking, verification, and lifecycle handoff files. -- Kept business code unchanged. - -## Verification Evidence - -| Check | Command | Result | Notes | -|---|---|---|---| -| Structural validation | `validate-harness.mjs --target .` | Passed | 100/100; 25/25 checks | -| Feature tracker | PowerShell JSON parse | Passed | Valid JSON | - -## Files Changed - -- `AGENTS.md` -- `feature_list.json` -- `progress.md` -- `init.sh` -- `session-handoff.md` - -## Decisions Made - -- Routine verification does not execute external model or search APIs. - -## Blockers / Risks - -- Project development tools require the configured Python environment; run `uv sync --extra dev` if needed. - -## Next Session Startup - -1. Read `AGENTS.md`, `feature_list.json`, and `progress.md`. -2. Review this handoff and `git status --short`. -3. Run `./init.sh` before editing. - -## Recommended Next Step - -- Add one concrete feature to `feature_list.json` before changing business code. diff --git a/skills-lock.json b/skills-lock.json deleted file mode 100644 index 033d35147..000000000 --- a/skills-lock.json +++ /dev/null @@ -1,11 +0,0 @@ -{ - "version": 1, - "skills": { - "harness-creator": { - "source": "walkinglabs/learn-harness-engineering", - "sourceType": "github", - "skillPath": "skills/harness-creator/SKILL.md", - "computedHash": "399b47cba9b61c43deedc165fe8bab8770da436a389de425b248fa8825a4fec3" - } - } -} From 9d65c474f4d455eaeeeb83196b4159e8a0e597de Mon Sep 17 00:00:00 2001 From: magic-ZSS <1165626853@qq.com> Date: Sun, 21 Jun 2026 02:41:33 +0800 Subject: [PATCH 4/5] =?UTF-8?q?$harness-creator=E4=B8=BA=E5=BD=93=E5=89=8D?= =?UTF-8?q?=E4=BB=93=E5=BA=93=E5=88=9D=E5=A7=8B=E5=8C=96=E6=9C=80=E5=B0=8F?= =?UTF-8?q?=E5=8F=AF=E7=94=A8=20Harness?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- AGENTS.md | 104 ++++++++++++++++++--------------------------- AGENTS.override.md | 81 +++++++++++++++++++++++++++++++++++ feature_list.json | 12 ++++++ init.sh | 19 +++++++++ progress.md | 48 +++++++++++++++++++++ session-handoff.md | 42 ++++++++++++++++++ 6 files changed, 243 insertions(+), 63 deletions(-) create mode 100644 AGENTS.override.md create mode 100644 feature_list.json create mode 100644 init.sh create mode 100644 progress.md create mode 100644 session-handoff.md diff --git a/AGENTS.md b/AGENTS.md index e1dc2115e..c068d44bd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,81 +1,59 @@ -# Open Deep Research 仓库概览 +# Open Deep Research 智能体工作规范 -## 项目描述 +## 启动流程（Startup Workflow） -Open Deep Research 是一个可配置、完全开源的深度研究智能体(deep research agent)，支持多个模型提供商、搜索工具以及 MCP（Model Context Protocol，模型上下文协议）服务器。它能够通过并行处理自动执行研究任务，并生成内容全面的研究报告。 +开始修改前： -## 仓库结构 +1. 确认位于仓库根目录并阅读 `README.md`。 +2. 阅读 `feature_list.json`、`progress.md` 和 `session-handoff.md`。 +3. 执行 `git status --short`，保留用户已有及无关改动。 +4. 一次只处理一个功能（One feature at a time）。 +5. 运行 `./init.sh`；若基线失败，先记录证据再修改。 -### 根目录 +缺少开发依赖时执行 `uv sync --extra dev`。 -* `README.md` - 完整的项目文档，包含快速入门指南 -* `pyproject.toml` - Python 项目配置与依赖声明 -* `langgraph.json` - LangGraph 配置文件，用于定义主图(Graph)的入口点 -* `uv.lock` - UV 包管理器的依赖锁定文件 -* `LICENSE` - MIT 许可证 -* `.env.example` - 环境变量模板（不受版本控制追踪） +## 项目范围（Scope） -### 核心实现（`src/open_deep_research/`） +核心代码位于 `src/open_deep_research/`，认证代码位于 `src/security/`， +评估工具位于 `tests/`，遗留实现位于 `src/legacy/`。 +`deep_research_from_scratch/` 是独立教学模块，除非当前功能明确要求，否则不要修改。 -* `deep_researcher.py` - LangGraph 的主要实现文件（入口点：`deep_researcher`） -* `configuration.py` - 配置管理与设置 -* `state.py` - 图状态(Graph state)定义与数据结构 -* `prompts.py` - 系统提示词与提示词模板 -* `utils.py` - 工具函数与辅助功能 -* `files/` - 研究输出文件与示例文件 +- 不得静默扩大功能范围。 +- 不得覆盖或清理无关工作树改动。 +- 不得提交 `.env`、API 密钥、私有 MCP 配置或敏感报告。 -### 遗留实现（`src/legacy/`） +## 状态文件 -包含两种早期的研究实现： +- `feature_list.json`：功能 status、dependencies 和 evidence 的唯一来源。 +- `progress.md`：记录当前状态、决定、阻塞项、文件和下一步。 +- `session-handoff.md`：跨会话恢复入口。 +- `init.sh`：统一验证入口。 -* `graph.py` - 带有人机协同(human-in-the-loop)机制的规划与执行(plan-and-execute)工作流 -* `multi_agent.py` - 监督者—研究员(supervisor-researcher)多智能体架构 -* `legacy.md` - 遗留实现的说明文档 -* `CLAUDE.md` - 面向遗留实现的 Claude 专用指令 -* `tests/` - 遗留实现专用测试 +状态仅使用 `not-started`、`in-progress`、`blocked`、`completed`。 -### 安全模块（`src/security/`） +## 验证命令（Verification Commands） -* `auth.py` - 用于 LangGraph 部署的身份验证处理程序 +```bash +./init.sh +``` -### 测试（`tests/`） +脚本执行源码编译、Ruff、mypy 和遗留测试收集。它不会运行 +`python tests/run_evaluate.py`，因为完整评估依赖外部服务并可能产生费用。 -* `run_evaluate.py` - 主评估脚本，配置为在 Deep Research Bench 上运行 -* `evaluators.py` - 专用评估函数 -* `prompts.py` - 评估提示词与评估标准 -* `pairwise_evaluation.py` - 对比评估工具 -* `supervisor_parallel_evaluation.py` - 多线程并行评估 +## 完成标准（Definition of Done） -### 示例（`examples/`） +功能只有同时满足以下条件才算完成： -* `arxiv.md` - ArXiv 研究示例 -* `pubmed.md` - PubMed 研究示例 -* `inference-market.md` - 推理市场分析示例 +- 范围内行为已实现。 +- 相关验证已运行并通过。 +- 命令结果或其他证据已记录。 +- 相关文档已更新。 +- 状态文件足以让下一会话独立恢复。 -### 手把手教学（deep_research_from_scratch/） -* 改文件夹下放的该项目进面向人类初学者的教学jupyter notebook内容，可视为与本项目的实际运行和使用完全解耦的模块 +## 会话结束（End of Session） -## 核心技术 - -* **LangGraph** - 工作流编排与图执行(Graph execution) -* **LangChain** - 大语言模型集成与工具调用(tool calling) -* **多个 LLM 提供商** - 支持 OpenAI、Anthropic、Google、Groq 和 DeepSeek -* **搜索 API** - 支持 Tavily、OpenAI/Anthropic 原生搜索、DuckDuckGo 和 Exa -* **MCP Servers** - 通过 Model Context Protocol 扩展智能体能力 - -## 开发命令 - -* `uvx langgraph dev` - 启动带有 LangGraph Studio 的开发服务器 -* `python tests/run_evaluate.py` - 运行完整评估 -* `ruff check` - 执行代码 lint 检查 -* `mypy` - 执行类型检查 - -## 配置 - -所有设置均可通过以下方式进行配置： - -* 环境变量（`.env` 文件） -* LangGraph Studio 中的 Web UI -* 直接修改配置文件 - -关键设置包括模型选择、搜索 API 选择、并发限制以及 MCP server 配置。 +1. 更新功能 status 和 evidence。 +2. 在 `progress.md` 记录验证、文件、阻塞项和下一步。 +3. 未完成时更新 `session-handoff.md`。 +4. 检查 `git status --short`。 +5. 留下 clean、可重复验证且不依赖聊天记录的恢复路径。 diff --git a/AGENTS.override.md b/AGENTS.override.md new file mode 100644 index 000000000..e1dc2115e --- /dev/null +++ b/AGENTS.override.md @@ -0,0 +1,81 @@ +# Open Deep Research 仓库概览 + +## 项目描述 + +Open Deep Research 是一个可配置、完全开源的深度研究智能体(deep research agent)，支持多个模型提供商、搜索工具以及 MCP（Model Context Protocol，模型上下文协议）服务器。它能够通过并行处理自动执行研究任务，并生成内容全面的研究报告。 + +## 仓库结构 + +### 根目录 + +* `README.md` - 完整的项目文档，包含快速入门指南 +* `pyproject.toml` - Python 项目配置与依赖声明 +* `langgraph.json` - LangGraph 配置文件，用于定义主图(Graph)的入口点 +* `uv.lock` - UV 包管理器的依赖锁定文件 +* `LICENSE` - MIT 许可证 +* `.env.example` - 环境变量模板（不受版本控制追踪） + +### 核心实现（`src/open_deep_research/`） + +* `deep_researcher.py` - LangGraph 的主要实现文件（入口点：`deep_researcher`） +* `configuration.py` - 配置管理与设置 +* `state.py` - 图状态(Graph state)定义与数据结构 +* `prompts.py` - 系统提示词与提示词模板 +* `utils.py` - 工具函数与辅助功能 +* `files/` - 研究输出文件与示例文件 + +### 遗留实现（`src/legacy/`） + +包含两种早期的研究实现： + +* `graph.py` - 带有人机协同(human-in-the-loop)机制的规划与执行(plan-and-execute)工作流 +* `multi_agent.py` - 监督者—研究员(supervisor-researcher)多智能体架构 +* `legacy.md` - 遗留实现的说明文档 +* `CLAUDE.md` - 面向遗留实现的 Claude 专用指令 +* `tests/` - 遗留实现专用测试 + +### 安全模块（`src/security/`） + +* `auth.py` - 用于 LangGraph 部署的身份验证处理程序 + +### 测试（`tests/`） + +* `run_evaluate.py` - 主评估脚本，配置为在 Deep Research Bench 上运行 +* `evaluators.py` - 专用评估函数 +* `prompts.py` - 评估提示词与评估标准 +* `pairwise_evaluation.py` - 对比评估工具 +* `supervisor_parallel_evaluation.py` - 多线程并行评估 + +### 示例（`examples/`） + +* `arxiv.md` - ArXiv 研究示例 +* `pubmed.md` - PubMed 研究示例 +* `inference-market.md` - 推理市场分析示例 + +### 手把手教学（deep_research_from_scratch/） +* 改文件夹下放的该项目进面向人类初学者的教学jupyter notebook内容，可视为与本项目的实际运行和使用完全解耦的模块 + +## 核心技术 + +* **LangGraph** - 工作流编排与图执行(Graph execution) +* **LangChain** - 大语言模型集成与工具调用(tool calling) +* **多个 LLM 提供商** - 支持 OpenAI、Anthropic、Google、Groq 和 DeepSeek +* **搜索 API** - 支持 Tavily、OpenAI/Anthropic 原生搜索、DuckDuckGo 和 Exa +* **MCP Servers** - 通过 Model Context Protocol 扩展智能体能力 + +## 开发命令 + +* `uvx langgraph dev` - 启动带有 LangGraph Studio 的开发服务器 +* `python tests/run_evaluate.py` - 运行完整评估 +* `ruff check` - 执行代码 lint 检查 +* `mypy` - 执行类型检查 + +## 配置 + +所有设置均可通过以下方式进行配置： + +* 环境变量（`.env` 文件） +* LangGraph Studio 中的 Web UI +* 直接修改配置文件 + +关键设置包括模型选择、搜索 API 选择、并发限制以及 MCP server 配置。 diff --git a/feature_list.json b/feature_list.json new file mode 100644 index 000000000..4cdaf7a44 --- /dev/null +++ b/feature_list.json @@ -0,0 +1,12 @@ +{ + "features": [ + { + "id": "harness-001", + "name": "初始化最小可用智能体 Harness", + "description": "在不修改业务代码的前提下，建立指令、状态、验证、范围控制和会话交接机制。", + "dependencies": [], + "status": "completed", + "evidence": "五个 Harness 文件已生成；结构验证 100/100（25/25），JSON 与格式检查通过，未修改业务代码。" + } + ] +} diff --git a/init.sh b/init.sh new file mode 100644 index 000000000..19e9cf8e5 --- /dev/null +++ b/init.sh @@ -0,0 +1,19 @@ +#!/bin/bash +set -e + +echo "=== Open Deep Research Harness 验证 ===" + +echo "=== 编译 Python 源码 ===" +python -m compileall -q src + +echo "=== Ruff 检查 ===" +python -m ruff check . + +echo "=== mypy 类型检查 ===" +python -m mypy src + +echo "=== 收集遗留测试（不调用外部 API）===" +python -m pytest --collect-only -q src/legacy/tests + +echo "=== 验证完成 ===" +echo "读取 feature_list.json，选择一个功能，并在 progress.md 记录证据。" diff --git a/progress.md b/progress.md new file mode 100644 index 000000000..fa346ea2b --- /dev/null +++ b/progress.md @@ -0,0 +1,48 @@ +# 会话进度记录 + +## 当前状态（Current State） + +**最后更新（Last Updated）：** 2026-06-21 +**当前功能：** `harness-001` +**状态：** 已完成 + +## 已完成（What's Done） + +- 创建五个最小 Harness 文件。 +- 配置不调用外部模型或搜索 API 的日常检查。 +- 保留业务代码和已有无关改动。 + +## 最终检查（What's In Progress） + +- 五个 Harness 子系统全部通过验证。 +- JSON、格式和 Git 变更范围检查通过。 + +## 下一步（What's Next） + +1. 后续修改业务代码前新增一个具体功能。 +2. 缺少工具时运行 `uv sync --extra dev`。 +3. 运行 `./init.sh` 并记录结果。 + +## 阻塞项与风险（Blockers / Risks） + +- Ruff、mypy 和 pytest 需要开发环境。 +- 遗留质量测试调用外部服务，因此 `init.sh` 只收集测试。 + +## 本次修改文件（Files Modified This Session） + +- `AGENTS.md` +- `feature_list.json` +- `progress.md` +- `init.sh` +- `session-handoff.md` + +## 验证证据（Verification Evidence） + +- 结构验证：`100/100`，共 `25/25` 项通过。 +- JSON 与格式检查：通过。 +- 业务文件变更范围检查：通过。 +- 业务代码测试：本次未修改业务代码，无需执行。 + +## 下次会话说明 + +读取全部状态文件并运行 `./init.sh` 后，再开始具体功能。 diff --git a/session-handoff.md b/session-handoff.md new file mode 100644 index 000000000..e364e90ce --- /dev/null +++ b/session-handoff.md @@ -0,0 +1,42 @@ +# 会话交接 + +## 当前目标（Current Objective） + +- 目标：初始化并验证最小可用智能体 Harness。 +- 当前状态：已完成，所有结构检查通过。 +- 分支 / 提交：当前工作树，未创建提交。 + +## 本次已完成 + +- 添加指令、状态、验证、范围和生命周期文件。 +- 保持业务代码不变。 + +## 验证证据 + +| 检查 | 命令 | 结果 | 备注 | +|---|---|---|---| +| 结构验证 | `validate-harness.mjs --target .` | 通过 | 100/100，25/25 项 | +| 功能清单 | JSON 解析与格式检查 | 通过 | JSON 有效，无尾随空格 | +| 变更范围 | Git 路径检查 | 通过 | 未修改业务代码或项目配置 | + +## 修改文件（Files Changed） + +- `AGENTS.md` +- `feature_list.json` +- `progress.md` +- `init.sh` +- `session-handoff.md` + +## 阻塞项与风险（Blockers / Risks） + +- 当前 Python 环境缺少 Ruff、mypy 和 pytest；运行完整 `init.sh` 前需执行 `uv sync --extra dev`。 + +## 下次会话启动（Next Session） + +1. 阅读 `AGENTS.md`、`feature_list.json` 和 `progress.md`。 +2. 查看本交接文件与 `git status --short`。 +3. 修改前运行 `./init.sh`。 + +## 建议下一步（Recommended Next Step） + +- 修改业务代码前，在 `feature_list.json` 新增一个具体功能。 From 9c5f365b9b5fd6e97240b0c9e9cfd68a4128968d Mon Sep 17 00:00:00 2001 From: magic-ZSS <1165626853@qq.com> Date: Sun, 21 Jun 2026 02:53:36 +0800 Subject: [PATCH 5/5] =?UTF-8?q?=E8=BF=BD=E5=8A=A0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- AGENTS.md | 65 +++++++++++++++++++++++++++++++++++++ AGENTS.override.md | 81 ---------------------------------------------- 2 files changed, 65 insertions(+), 81 deletions(-) delete mode 100644 AGENTS.override.md diff --git a/AGENTS.md b/AGENTS.md index c068d44bd..1672168ed 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,3 +1,68 @@ +# Open Deep Research 仓库概览 + +## 项目描述 + +Open Deep Research 是一个可配置、完全开源的深度研究智能体(deep research agent)，支持多个模型提供商、搜索工具以及 MCP（Model Context Protocol，模型上下文协议）服务器。它能够通过并行处理自动执行研究任务，并生成内容全面的研究报告。 + +## 仓库结构 + +### 根目录 + +* `README.md` - 完整的项目文档，包含快速入门指南 +* `pyproject.toml` - Python 项目配置与依赖声明 +* `langgraph.json` - LangGraph 配置文件，用于定义主图(Graph)的入口点 +* `uv.lock` - UV 包管理器的依赖锁定文件 +* `LICENSE` - MIT 许可证 +* `.env.example` - 环境变量模板（不受版本控制追踪） + +### 核心实现（`src/open_deep_research/`） + +* `deep_researcher.py` - LangGraph 的主要实现文件（入口点：`deep_researcher`） +* `configuration.py` - 配置管理与设置 +* `state.py` - 图状态(Graph state)定义与数据结构 +* `prompts.py` - 系统提示词与提示词模板 +* `utils.py` - 工具函数与辅助功能 +* `files/` - 研究输出文件与示例文件 + +### 遗留实现（`src/legacy/`） + +包含两种早期的研究实现： + +* `graph.py` - 带有人机协同(human-in-the-loop)机制的规划与执行(plan-and-execute)工作流 +* `multi_agent.py` - 监督者—研究员(supervisor-researcher)多智能体架构 +* `legacy.md` - 遗留实现的说明文档 +* `CLAUDE.md` - 面向遗留实现的 Claude 专用指令 +* `tests/` - 遗留实现专用测试 + +### 安全模块（`src/security/`） + +* `auth.py` - 用于 LangGraph 部署的身份验证处理程序 + +### 测试（`tests/`） + +* `run_evaluate.py` - 主评估脚本，配置为在 Deep Research Bench 上运行 +* `evaluators.py` - 专用评估函数 +* `prompts.py` - 评估提示词与评估标准 +* `pairwise_evaluation.py` - 对比评估工具 +* `supervisor_parallel_evaluation.py` - 多线程并行评估 + +### 示例（`examples/`） + +* `arxiv.md` - ArXiv 研究示例 +* `pubmed.md` - PubMed 研究示例 +* `inference-market.md` - 推理市场分析示例 + +### 手把手教学（deep_research_from_scratch/） +* 改文件夹下放的该项目进面向人类初学者的教学jupyter notebook内容，可视为与本项目的实际运行和使用完全解耦的模块 + +## 核心技术 + +* **LangGraph** - 工作流编排与图执行(Graph execution) +* **LangChain** - 大语言模型集成与工具调用(tool calling) +* **多个 LLM 提供商** - 支持 OpenAI、Anthropic、Google、Groq 和 DeepSeek +* **搜索 API** - 支持 Tavily、OpenAI/Anthropic 原生搜索、DuckDuckGo 和 Exa +* **MCP Servers** - 通过 Model Context Protocol 扩展智能体能力 + # Open Deep Research 智能体工作规范 ## 启动流程（Startup Workflow） diff --git a/AGENTS.override.md b/AGENTS.override.md deleted file mode 100644 index e1dc2115e..000000000 --- a/AGENTS.override.md +++ /dev/null @@ -1,81 +0,0 @@ -# Open Deep Research 仓库概览 - -## 项目描述 - -Open Deep Research 是一个可配置、完全开源的深度研究智能体(deep research agent)，支持多个模型提供商、搜索工具以及 MCP（Model Context Protocol，模型上下文协议）服务器。它能够通过并行处理自动执行研究任务，并生成内容全面的研究报告。 - -## 仓库结构 - -### 根目录 - -* `README.md` - 完整的项目文档，包含快速入门指南 -* `pyproject.toml` - Python 项目配置与依赖声明 -* `langgraph.json` - LangGraph 配置文件，用于定义主图(Graph)的入口点 -* `uv.lock` - UV 包管理器的依赖锁定文件 -* `LICENSE` - MIT 许可证 -* `.env.example` - 环境变量模板（不受版本控制追踪） - -### 核心实现（`src/open_deep_research/`） - -* `deep_researcher.py` - LangGraph 的主要实现文件（入口点：`deep_researcher`） -* `configuration.py` - 配置管理与设置 -* `state.py` - 图状态(Graph state)定义与数据结构 -* `prompts.py` - 系统提示词与提示词模板 -* `utils.py` - 工具函数与辅助功能 -* `files/` - 研究输出文件与示例文件 - -### 遗留实现（`src/legacy/`） - -包含两种早期的研究实现： - -* `graph.py` - 带有人机协同(human-in-the-loop)机制的规划与执行(plan-and-execute)工作流 -* `multi_agent.py` - 监督者—研究员(supervisor-researcher)多智能体架构 -* `legacy.md` - 遗留实现的说明文档 -* `CLAUDE.md` - 面向遗留实现的 Claude 专用指令 -* `tests/` - 遗留实现专用测试 - -### 安全模块（`src/security/`） - -* `auth.py` - 用于 LangGraph 部署的身份验证处理程序 - -### 测试（`tests/`） - -* `run_evaluate.py` - 主评估脚本，配置为在 Deep Research Bench 上运行 -* `evaluators.py` - 专用评估函数 -* `prompts.py` - 评估提示词与评估标准 -* `pairwise_evaluation.py` - 对比评估工具 -* `supervisor_parallel_evaluation.py` - 多线程并行评估 - -### 示例（`examples/`） - -* `arxiv.md` - ArXiv 研究示例 -* `pubmed.md` - PubMed 研究示例 -* `inference-market.md` - 推理市场分析示例 - -### 手把手教学（deep_research_from_scratch/） -* 改文件夹下放的该项目进面向人类初学者的教学jupyter notebook内容，可视为与本项目的实际运行和使用完全解耦的模块 - -## 核心技术 - -* **LangGraph** - 工作流编排与图执行(Graph execution) -* **LangChain** - 大语言模型集成与工具调用(tool calling) -* **多个 LLM 提供商** - 支持 OpenAI、Anthropic、Google、Groq 和 DeepSeek -* **搜索 API** - 支持 Tavily、OpenAI/Anthropic 原生搜索、DuckDuckGo 和 Exa -* **MCP Servers** - 通过 Model Context Protocol 扩展智能体能力 - -## 开发命令 - -* `uvx langgraph dev` - 启动带有 LangGraph Studio 的开发服务器 -* `python tests/run_evaluate.py` - 运行完整评估 -* `ruff check` - 执行代码 lint 检查 -* `mypy` - 执行类型检查 - -## 配置 - -所有设置均可通过以下方式进行配置： - -* 环境变量（`.env` 文件） -* LangGraph Studio 中的 Web UI -* 直接修改配置文件 - -关键设置包括模型选择、搜索 API 选择、并发限制以及 MCP server 配置。