[prompt-clustering] Prompt Clustering Analysis - 2026-08-12

[github-actions[bot]]

Summary

Analysis Period: 2026-07-28 to 2026-08-12 (last 30 days)
Total Tasks Analyzed: 1,000 copilot-authored PRs (744 merged, 248 closed, 8 open)
Clusters Identified: 4
Overall Merge Rate: 74.4%

Limitation: gh-aw logs was not available in this run (no aw_info.json workflow-run data), so turn counts, duration, and cost could not be enriched per PR. This analysis relies on PR body/title text, comments, reviews, and file-change stats only.

Full Analysis Report

General Insights

• Most Common Task Type: General bug fixes, refactors, and test-coverage work (Cluster 2 — 515 PRs, 51.5% of all tasks)
• Highest Success Rate: Cluster 2, general bug/refactor/test work, at 79.4% merged
• Lowest Success Rate / Outlier: Cluster 3, container image & CVE remediation, at 53.9% merged — 20.5 points below the overall rate
• Most Complex Tasks: Cluster 1 (docs/AI-crawler/misc) and Cluster 3 (security) both average the largest diffs (~275–900 additions), but Cluster 3 also has the highest average files-changed (33.9)
• Outliers: 1 cluster met the outlier threshold (≥15 PRs, merge rate ≥10 points below overall)

Cluster Analysis

Cluster 1: General bug/refactor/test work

• Size: 515 tasks (51.5% of total)
• Success Rate: 79.4%
• Avg files changed / comments: 19.9 / 6.6
• Top Keywords: added, changes, test, tests, add, existing, new, coverage
• Top PR-type labels: bug (85), refactor (51), chore (25)
• Characteristics: Day-to-day engineering — bug fixes, refactors, and added test coverage across pkg/*. Highest volume and highest success rate of any cluster.
• Example PRs: refactor: replace string += in loops with strings.Builder (41 findings) #48484, refactor: introduce named type LLMProvider string to replace bare string threading #48485, refactor: replace duplicate experimentStateJSON/experimentRunRecord with exported ExperimentState/ExperimentRunRecord #48486, fix: propagate download errors through StartDockerImageDownload join function #48487

"refactor: replace string += in loops with strings.Builder (41 findings)"

---

Cluster 2: Workflow/agent infra & CI stabilization

• Size: 374 tasks (37.4% of total)
• Success Rate: 71.7%
• Avg files changed / comments: 31.0 / 4.4
• Top Keywords: workflow, run, step, workflows, job, agent, generated, failure
• Top PR-type labels: bug (38), chore (24), feature (14)
• Characteristics: Fixes and features touching the agentic-workflow engine itself — lock-file generation, smoke-test stabilization, sub-agent routing, prompt-bloat reduction. Second largest cluster, moderate success rate, largest average diffs by file count.
• Example PRs: Reduce ambient prompt bloat in daily workflow specs (script extraction, DIFC chart consolidation, proxy mode) #48483, Stabilize Daily Agent of the Day sub-agent model routing for Copilot chat endpoint compatibility #48504, [WIP] Fix sub agent status in smoke tests #48512, Stabilize smoke-copilot-auto lock output by skipping dynamic auto pricing injection #48516

"Stabilize Daily Agent of the Day sub-agent model routing for Copilot chat endpoint compatibility"

---

Cluster 3: Container image / CVE remediation ⚠️ outlier

• Size: 76 tasks (7.6% of total)
• Success Rate: 53.9% (vs. 74.4% overall — a 20.5-point gap)
• Avg files changed / comments: 33.9 / 3.6
• Top Keywords: image, container, digest, high, pinned, scan, cve, mcp
• Top PR-type labels: chore (11), bug (10)
• Characteristics: Automated remediation of container-image CVEs, digest pins, and license-scan findings — largely triggered by recurring security-scan workflows rather than one-off human requests. See root-cause section below.
• Example PRs: Strengthen actionpins spec tests for cache keys, fallback auditing, and container digest validation #48497, [WIP] Fix vulnerabilities in ghcr.io/github/gh-aw-firewall/api-proxy #48498, fix: update brace-expansion to 5.0.8 and upgrade Alpine base image to 3.22 #48521, Remove vulnerable gh-aw-firewall/squid:0.27.41 pin from embedded container pin catalogs #48534

"Remove vulnerable gh-aw-firewall/squid:0.27.41 pin from embedded container pin catalogs"

---

Cluster 4: Docs, AI-crawler policy & misc engine work

• Size: 35 tasks (3.5% of total)
• Success Rate: 74.3%
• Avg files changed / comments: 39.5 / 4.4
• Top Keywords: ai, agent, run, docs, audit, engine, file, changes
• Top PR-type labels: bug (2), docs (1), feature (1)
• Characteristics: A small, mixed bag — robots.txt/llms.txt AI-crawler policy updates, engine-version normalization, and one-off investigations. Success rate tracks the overall average.
• Example PRs: feat(docs): add robots.txt with AI-bot Allow rules for docs site #48697, Fixing model name normalizer logic #49001, feat(docs): update llms.txt to GEO-compliant spec with full docs site index #49177, Analyzing agentic_maintenance.yml generation issues #50107

"feat(docs): add robots.txt with AI-bot Allow rules for docs site"

Success Rate by Cluster

Cluster	Tasks	Success Rate	Avg Files Changed	Avg Comments	Top Keywords
General bug/refactor/test	515	79.4%	19.9	6.6	added, test, refactor, coverage
Workflow/agent infra & CI	374	71.7%	31.0	4.4	workflow, run, job, failure
Container/CVE remediation	76	53.9%	33.9	3.6	image, container, cve, scan
Docs & misc	35	74.3%	39.5	4.4	docs, ai, engine, audit

Turn counts and cost are omitted — workflow-run logs were unavailable this run (see limitation above).

Sample PRs per cluster (20 of 1,000 total — full table omitted for length)

PR #	Title	Cluster	Outcome	Comments
#48484	refactor: replace string += in loops with strings.Builder	General bug/refactor/test	Closed	0
#48485	refactor: introduce named type LLMProvider string	General bug/refactor/test	Merged	5
#48486	refactor: replace duplicate experimentStateJSON/experimentRunRecord	General bug/refactor/test	Merged	5
#48487	fix: propagate download errors through StartDockerImageDownload	General bug/refactor/test	Merged	2
#48483	Reduce ambient prompt bloat in daily workflow specs	Workflow/agent infra & CI	Closed	1
#48512	[WIP] Fix sub agent status in smoke tests	Workflow/agent infra & CI	Closed	—
#48516	Stabilize smoke-copilot-auto lock output	Workflow/agent infra & CI	Merged	—
#48497	Strengthen actionpins spec tests for cache keys, container digest validation	Container/CVE remediation	Merged	—
#48521	fix: update brace-expansion to 5.0.8 and upgrade Alpine base image	Container/CVE remediation	Merged	—
#48534	Remove vulnerable gh-aw-firewall/squid:0.27.41 pin	Container/CVE remediation	Closed (dup, 33 min)	0
#51770	Bump gh-aw-mcpg image to v0.4.9	Container/CVE remediation	Closed (dup)	1
#51796	Bump gh-aw-mcpg to v0.4.9	Container/CVE remediation	Closed (dup)	0
#51828	Bump gh-aw-mcpg to v0.4.9 and github-mcp-server to v1.9.0	Container/CVE remediation	Merged	—
#48697	feat(docs): add robots.txt with AI-bot Allow rules for docs site	Docs & misc	Merged	—
#49177	feat(docs): update llms.txt to GEO-compliant spec	Docs & misc	Merged	—

Key Findings

1. General engineering work dominates and succeeds most often. Cluster 1 (bug fixes, refactors, added tests) is over half of all tasks and merges at 79.4% — this is the bread-and-butter, low-friction workload for the agent.
2. Workflow/infra tasks touch the most files but merge less often. Cluster 2 averages 31 files changed per PR (vs. 19.9 for general work), consistent with cross-cutting changes to lock files and workflow specs — larger blast radius likely explains the somewhat lower 71.7% success rate.
3. Container/CVE remediation is a clear negative outlier at 53.9% merge rate, driven almost entirely by duplicate/superseded PRs and abandoned drafts rather than by review rejection or CI failure — see root-cause analysis below.

Lowest-Merge Cluster Root Cause

• Cluster: Container image / CVE remediation (76 PRs)
• Merge Rate Gap: 53.9% vs. 74.4% overall (gap: 20.5 points)
• Primary Blocker: Redundant, concurrently-filed remediation PRs (multiple agent runs racing to fix the same CVE/image) plus abandoned [WIP] drafts — not CI failures or reviewer rejection
• Confidence: High
• Evidence PRs: Remove vulnerable gh-aw-firewall/squid:0.27.41 pin from embedded container pin catalogs #48534, Block scan remediation until patched gh-aw-mcpg image is published #49797, Bump gh-aw-mcpg image to v0.4.9 #51770, Bump gh-aw-mcpg to v0.4.9 #51796 (duplicates of Bump gh-aw-mcpg to v0.4.9 and github-mcp-server to v1.9.0 #51828, all closed), [WIP] Fix container image vulnerabilities for ghcr.io/github/gh-aw-mcpg:v0.4.6 #49672, [WIP] Update container image mcp/arxiv-mcp-server for vulnerabilities #49838 (abandoned [WIP] drafts)

Of the 35 closed (non-merged) PRs in this cluster:

Blocker Category	Count	Share	Notes
Duplicate/concurrent remediation	20	57%	Closed with 0–1 comments, median time-to-close ~30 min; e.g. three separate PRs (#51754 merged, #51770 and #51796 closed) all titled "Bump gh-aw-mcpg to v0.4.9"
Abandoned [WIP] draft	14	40%	Title prefixed [WIP], task apparently too large/blocked to complete in one run (e.g. waiting on an upstream patched image)
Merge conflict / CI	1	3%	Explicit "@copilot resolve the merge conflicts" comment

Merged PRs in this cluster average 6.4 comments each; closed PRs average 0.4, and 80% of closed PRs (28/35) have zero comments at all — a strong signal these are being superseded/duplicated rather than substantively reviewed and rejected. This pattern (quick, low-engagement closes) exists dataset-wide too (~51–60% of all closed PRs), but is markedly stronger in this cluster, consistent with recurring daily security-scan workflows independently filing near-identical fixes for the same vulnerability before the first fix merges.

Recommendations

1. Deduplicate security-scan remediation PRs before creation. The container/CVE workflow(s) should check for an existing open PR targeting the same image/CVE/digest before filing a new one — this alone would likely eliminate over half of this cluster's closed PRs (the "Bump gh-aw-mcpg to v0.4.9" triple-filing is a concrete example).
2. Add an explicit "blocked on upstream" state for CVE PRs that can't be completed, instead of leaving them as [WIP] drafts that get silently closed — e.g. auto-label and comment with the blocking dependency, then re-trigger once resolved.
3. Keep investing in the general bug/refactor/test workflow (Cluster 1) — it's the highest-volume, highest-success pattern; no changes needed there.
4. Container/CVE remediation cluster: gate the recurring scan workflow so it only opens a new PR when no equivalent PR is already open for that image/finding, and add a lightweight "supersedes #N" auto-comment + auto-close when a newer fix lands, so duplicate closures stop looking like silent failures in future audits.

---

Generated by Prompt Clustering Analysis (Run: §31585642928)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · agent · 107 AIC · ⌖ 51.3 AIC · ⊞ 14.5K · ◷

• expires on Aug 13, 2026, 2:31 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #52476.