Skip to content

feat(require-annotation-reason): require a reason on skip and fixme - #497

Open
unlikelyzero wants to merge 1 commit into
mskelton:mainfrom
unlikelyzero:feat/require-annotation-reason
Open

feat(require-annotation-reason): require a reason on skip and fixme#497
unlikelyzero wants to merge 1 commit into
mskelton:mainfrom
unlikelyzero:feat/require-annotation-reason

Conversation

@unlikelyzero

Copy link
Copy Markdown
Collaborator

Adds a require-annotation-reason rule.

The enforceable norm on most teams isn't "never skip a test" — it's "never skip a test silently." test.skip(isMobile) tells the next reader nothing: six months later nobody knows whether the underlying bug was fixed, whether the condition still holds, or who to ask, so the annotation stays forever and the coverage never comes back. test.skip(isMobile, 'ref WET-204 — layout breaks below 768px') costs nothing and answers all three.

That's a norm a team can turn on at error today, which is not true of a blanket prohibition on skipping. It came out of running no-skipped-test against a mature suite where every fixme already carried a Jira key — enforced purely by convention and code review.

Scope

The rule checks the conditional form, where description is a documented Playwright parameter:

test.skip(condition, description)

skip and fixme are checked by default; fail and slow don't remove coverage the same way, so they're opt-in via annotations. An empty or whitespace-only reason counts as missing. A reason whose value isn't known statically (a call, an interpolated template) is accepted as present but not pattern-checked.

The declaration form (test.skip('title', fn)) is deliberately not reported — Playwright's signature has no description parameter there, so there'd be no way to satisfy the rule. That case belongs to no-skipped-test, and the two rules are complementary and independently adoptable: this one governs how an annotation is written, that one governs whether it may be used at all.

pattern

An optional regex the reason must match, which is what makes the ticket-reference convention enforceable in CI rather than in review:

{
  "playwright/require-annotation-reason": ["error", { "pattern": "\\b[A-Z]+-\\d+\\b" }]
}
test.fixme(isMobile, 'ref WET-204')  // ok
test.fixme(isMobile, 'flaky')        // error
test.fixme(isMobile)                 // error

Not enabled in the recommended config.

Note

This branches off main, so testInfo.skip(...) / testInfo.fixme(...) aren't recognised yet. It picks that up for free once #490 lands, since detection lives in parseFnCall.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant