Skip to content

feat: provider discovery as a shipped tool - engine, CLI, MCP tools, JSON entries - #10

Open
MrJ55 wants to merge 10 commits into
hanzili:mainfrom
MrJ55:feat/discovery-as-tool
Open

feat: provider discovery as a shipped tool - engine, CLI, MCP tools, JSON entries#10
MrJ55 wants to merge 10 commits into
hanzili:mainfrom
MrJ55:feat/discovery-as-tool

Conversation

@MrJ55

@MrJ55 MrJ55 commented Aug 7, 2026

Copy link
Copy Markdown

Summary

Makes provider discovery a shipped, triggerable capability of comet-mcp instead of a test artifact, and adds self-healing provider controls (ADR 0003) so the harness is the rare repair, not the routine check.

Two capabilities:

  1. Discovery as a shipped tool — when a provider changes its DOM and comet-mcp breaks, re-running discovery is repeatable and on-demand. Provider entries are data (JSON), not code, written directly by the discovery engine — so DOM-drift repair is discover → commit new JSON, no code changes.

  2. Self-healing controls (ADR 0003) — inspired by Bladebro's Live Page Model (src/knowledge.rs, src/page/refs.rs). provider_verify is now a learning loop: per-control confidence (success +0.05, failure −0.15 asymmetric, evict <0.3, trust ≥0.7 hot-path resolve, learn-only-from-success). On a selector miss, a structural fingerprint rebind (in-page FNV-1a of ancestor chain + tag + children + identity attrs) absorbs React/Vue re-renders before anything escalates — only a genuine DOM change degrades and suggests provider_discover --diff.

Base note: this branch is stacked on feat/fabric-types-p0-p2 (PR #9) because the engine depends on the P1 canonical types landed there. It is open against main; merge order #8#9#10 and GitHub auto-shrinks the diff to this PR's delta.

What's included

src/core/discovery.ts — the engine (ported from the discovery harness):

  • runDiscovery(provider, {write, deadlineMs}) — inventory → composer/send detection → one varied validation prompt → streaming/completed observation → writes entry JSON + fixtures
  • verifyProvider(provider)cheap selector check, sends NO prompt (the drift gate); now a confidence-learning loop with fingerprint rebind on miss
  • diffEntry(provider, next) — selector/heuristic changes vs the committed entry
  • pickPrompt — per-run prompt rotation (persisted state) so no provider repeats a probe prompt
  • Zero dependencies (Node native WebSocket); own CDP connection per run

src/core/registry.ts — loads + validates src/providers/entries/*.json; confidence model (recordSuccess/recordFailure/resolveWithConfidence/persistControlUpdate); packageRoot() walk-up so paths work from compiled dist/.

src/core/fingerprint.ts — in-page FNV-1a structural fingerprint (ancestor chain ≤10 + tag + first-3 children + type/name/data-testid, per Bladebro perception.rs) + resolveWithRebind: known selector → fingerprint rebind on miss → null escalates to discovery. Resolution order: known → fingerprint-rebind → heuristic → discovery.

src/cli.ts + bin dispatch — the primary on-demand trigger:

comet-mcp discover --provider grok [--diff] [--no-write]
comet-mcp verify   --provider grok        # no prompt sent; reports per-control confidence + rebinds
comet-mcp list

MCP tools (provider_discover, provider_verify) — opt-in operational tools per ADR 0001 (discovery is not a hot-path dependency).

Provider entries as JSONsrc/providers/entries/{perplexity,grok,gemini,chatgpt,claude}.json, all HIGH confidence from live verification (2026-08-06/07, ACK/PONG/ALPHA/OK/BRAVO validations). Each control carries confidence/fingerprint fields; conditional controls (send buttons rendered only after typing) are flagged and skipped by idle verify. The hand-written .ts entries are retired.

Docs — ADR 0003 (self-healing provider controls); build plan phase status (P0 done, P1/P2 types+discovery done, runtime pending); generalized discovery runbook; Turn-02 checklist updated.

Fixturestest/fixtures/{perplexity,gemini,chatgpt,claude}/ state snapshots.

Notes for reviewers

  • tsc --noEmit + npm run build pass.
  • Verified live: comet-mcp verify --provider perplexity → HEALTHY with confidence climbing (0.70→0.80 across verifies) and fingerprints captured; verify --provider gemini → HEALTHY (conditional sendButton correctly skipped, unconditional controls at 1.00); verify --provider grok gracefully reports "no grok tab found" (exit 1); plain comet-mcp still starts the server.
  • Conditional-control fix (caught live): send buttons rendered only after typing were initially counted as verify failures at idle (false negative — Gemini's sendButton drained 0.9→0.6 in two verifies). Fixed: conditional controls are skipped by idle verify (no confidence penalty, not a health failure); their precondition is exercised by provider_discover.
  • The discovery engine sends exactly one validation prompt per run, rotated per provider (ACK/ALPHA/OK/BRAVO/PONG pool) — repeated identical probes would look like an injection sweep.
  • Legacy test/integration/discover.mjs harness removed (superseded by the engine).
  • Rotation state + test/integration/out/ are gitignored as runtime artifacts.

Drift workflow this enables

comet-mcp verify --provider grok              # MISS → confidence drops; re-renders rebind silently
comet-mcp discover --provider grok --diff     # only on genuine DOM change / eviction
git diff src/providers/entries/grok.json      # review selector changes
git commit                                     # repaired — no code changes

Related

MrJ55 added 10 commits August 6, 2026 10:09
getAgentStatus truncated long answers to their final fragment and could get
stuck reporting WORKING after the answer finished streaming.

Fixes:
- Join ALL [class*="prose"] blocks with containment dedup instead of taking
  only the last element - long answers are returned in full, without
  duplicated nested content.
- Raise the response cap from 8000 to 30000 chars.
- Completion detection: "Ask a follow-up" + prose wins over the
  working-text heuristic, so answers whose text contains words like
  "Working"/"Searching"/"Analyzing" no longer leave comet_poll stuck on
  WORKING.
- Harden getAgentStatus against undefined CDP evaluate results (dead tab,
  navigate race, closed browser) so callers never crash on status.toUpperCase().

Verified end-to-end against live Perplexity threads (search mode) through the
MCP gateway: comet_poll returns the complete, deduplicated answer.
…ry types

Port the discovery harness into the tool: src/core/discovery.ts (runDiscovery,
verifyProvider, diffEntry, pickPrompt with per-run prompt rotation) and
src/core/registry.ts (load/validate ProviderEntry JSON, packageRoot path
resolution). Canonical ProviderEntry/ProviderControl types move into
src/types/provider.ts so entries are data, not code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant