Skip to content

Uses the project-discipline-guides SKILL to remove "common" knowledge from discipline guides - #808

Open
micahjo7 wants to merge 1 commit into
mainfrom
discipline-guides-cutdown
Open

Uses the project-discipline-guides SKILL to remove "common" knowledge from discipline guides#808
micahjo7 wants to merge 1 commit into
mainfrom
discipline-guides-cutdown

Conversation

@micahjo7

@micahjo7 micahjo7 commented May 14, 2026

Copy link
Copy Markdown
Collaborator

The SKILL was developed in #748.

  1. First it pulls content generated from Claude Code, Gemini CLI, Codex CLI regarding a "discipline".
  2. It combines the common information from each of the three into one mirror.md file.
  3. This mirror.md file content is compared with the discipline guide (e.g. guides/accessibility/accessibility/guide.md)
  4. The content from the guide that is already in the mirror is removed from the guide.

This way we ensure that the guides do not contain information that (most) coding agents already have know.

This PR shows the results of running this SKILL on all the discipline guides that we have checked in currently.
Note that the webmcp guide did not have any changes (no common knowledge found).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline, I'm uncomfortable to wholesale remove content from these guides without more data first

@LeaVerou

LeaVerou commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

I’ll need to preface this with the disclaimer that I've mainly looked at the css, html, and forms guides but from a quick look this doesn't seem to work very well, at least for those… 😕

It removed many things that agents don't actually do in practice and merged sentences in ways that completely changed the meaning (e.g. the bit about :where() that now became conditional to layer usage).

I hate to be a party pooper, especially having said for months that we need to remove guidance that agents already follow, which is exactly what this does! I love the idea, but I suspect the approach has a fundamental flaw: If you ask agents about best practices directly they will give you a lot of things they don't actually apply in practice because knowing about a feature and actually using it are different things.

I don't have a concrete counter-proposal but I think to remove guidance from hand-authored guides in an automated way, we'd need to be testing that that guidance being removed is actually followed by agents, not just known. …Which is basically what the current evals system does. Perhaps what we'd need is a very lightweight version of that. 🤔

I think this can still be useful:

  • to point to as claims about what agents do and don't know often come up in discussions and these mirrors are a useful lower bound.
  • The bits removed are also a useful signal, but need to be evaluated carefully by ATLs, one by one rather than merged wholesale, IMO.
  • To remove content from guides that have been AI-generated and not hand-authored (such as the initial versions of discipline skills)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants