Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ Nuxt file-based routing:
- **Failure clustering / fingerprints**: The grouping key is computed in `shared/error-fingerprint.ts` (`computeErrorFingerprint`) over error type + normalized message + masked locator — the **stack frame is intentionally NOT hashed** (kept as `topFrameFile` for display only) so the same root cause groups across spec files. Add new volatile-token masking inside `maskVolatile` (message) / `maskSelector` (locator). When you change normalization, bump `FINGERPRINT_VERSION` and keep the demo mirror in `scripts/generate-demo-seed.mjs#computeDemoFingerprint` in sync. A version bump is **non-destructive**: `reclusterFailureFingerprints()` (`shared/handlers/failure-cluster-recluster.ts`) re-fingerprints existing clusters from their stored `sampleError` on startup, updating in place or merging collisions via `mergeFailureClusters()` (`shared/handlers/failure-cluster-ops.ts`) so triage state survives. Re-run `npm run app:seed:demo` after any change.
- **Spec health**: Endpoint `GET /api/projects/[id]/spec-health` groups by spec prefix. `SpecHealthTable.vue` renders a sortable `UTable` (pass rate, flaky rate, failures, tests, avg time per spec prefix; pass-rate dot + linked prefix). Add `spec-health` to `validTabs` and `tabItems` in project `index.vue` page.
- **API docs live in the in-app Scalar UI, not in VitePress docs**: The auto-generated OpenAPI spec (`/_openapi.json`) and interactive reference (`/docs`) in the running app or demo are the single source of truth for API documentation. There is no `docs/api.md` file in the VitePress docs site. Instead, the VitePress nav/sidebar links to the demo's `/docs` page (`https://piwitests.github.io/demo/docs`). When documenting features in VitePress `.md` files, reference `[API docs](/docs)` (for self-hosted) or the live demo link rather than inline endpoint descriptions, and never create a static API reference page.
- **Locator healing**: Capture happens in the reporter fixtures (`reporter/src/internal/capture/capture-fixtures.ts`, mirrored by the dogfood `application/tests/fixtures.ts`): a `Proxy` wraps the page-level locator methods in `LOCATOR_METHODS` and, after each successful action, records the element's attributes plus ranked alternatives (`generateAlternatives` in `reporter/src/internal/capture/locator-healing.ts`) stamped with `captureCallerLocation()`. The snapshots ride the wire as the transient `locatorSnapshots` per-case field (`shared/types.ts`, never a column) → every ingest site (`submit`, `upload`, `[id]/events`, + the demo `app/demo/api/reporter.ts` mirror) passes it to `persistRunCases` → the **shared** `upsertLocatorSnapshots` (`server/utils/locator-healing.ts`, imported by both server and demo) upserts one row per `(test_case_id, location)` into `locator_snapshots`. **Two invariants live in that helper:** rows are deduped by `(caseId, location)` before the batch upsert (a repeated call site would otherwise make PostgreSQL reject the `ON CONFLICT DO UPDATE`), and the stale-location purge runs **only for cases whose run passed** (`purge` flag) so a run that failed before reaching later locators doesn't delete their valid prior-success rows. Lookup (`getLocatorHealing`) runs a ladder: exact call-site location → `file:line` → locator signature → element-match against the failing ARIA snapshot → ARIA fallback. The locator signature must hash identically on both sides — capture via `locatorSignature(method, args)` and lookup via `locatorSignatureFromExpression(expr)` (`shared/locator-healing.ts`); the error's leaf selector is extracted by `extractLeafSelector` (`shared/error-fingerprint.ts`). Pure helpers live in `shared/` (`locator-healing.ts`, `locator-fingerprint.ts`, `locator-healing.types.ts`); the reporter keeps a structural mirror (no `shared` import). Toggle with the `captureLocators` reporter option / `PIWI_CAPTURE_LOCATORS` env var (auto-off when `collectPerformanceMetrics` is false). Surfaced via `LocatorHealingPanel.vue` and the `locatorHealing` AI-diagnosis section. Re-run `npm run app:seed:demo` after changing the captured/stored shape.
- **Locator healing**: Capture happens in the reporter fixtures (`reporter/src/internal/capture/capture-fixtures.ts`, mirrored by the dogfood `application/tests/fixtures.ts`): a `Proxy` wraps the page-level locator methods in `LOCATOR_METHODS` and, after each successful action, records the element's attributes plus ranked alternatives (`generateAlternatives` in `reporter/src/internal/capture/locator-healing.ts`) stamped with `captureCallerLocation()`. The in-page probe also reports `hasLabel` (gates `getByLabel` — an accessible name approximated from placeholder/title must not produce one) and `selectorCounts` (querySelectorAll uniqueness counts; ambiguous testid/id/name/class alternatives are dropped). The live input `value` is deliberately never captured (secret leak). Snapshots are deduped by location (`dedupeSnapshotsByLocation`) before attaching. The snapshots ride the wire as the transient `locatorSnapshots` per-case field (`shared/types.ts`, never a column) → every ingest site (`submit`, `upload`, `[id]/events`, + the demo `app/demo/api/reporter.ts` mirror) passes it to `persistRunCases` → the **shared** `upsertLocatorSnapshots` (`server/utils/locator-healing.ts`, imported by both server and demo) upserts one row per `(test_case_id, location)` into `locator_snapshots`. **Two invariants live in that helper:** rows are deduped by `(caseId, location)` before the batch upsert (a repeated call site would otherwise make PostgreSQL reject the `ON CONFLICT DO UPDATE`), and the stale-location purge runs **only for cases whose run passed** (`purge` flag) so a run that failed before reaching later locators doesn't delete their valid prior-success rows. Lookup (`getLocatorHealing`) runs a ladder: exact call-site location → `file:line` (path-suffix tolerant) → locator signature → cross-test (same signature captured by another test in the project, freshest wins) → element-match against the failing ARIA snapshot → ARIA fallback. `recommendation.suggestAddTestId` is only meaningful for stability-scored sources (`prior-run`/`fingerprint`/`cross-test`) and is force-cleared otherwise. The locator signature must hash identically on both sides — capture via `locatorSignature(method, args)` and lookup via `locatorSignatureFromExpression(expr)` (`shared/locator-healing.ts`); the error's leaf selector is extracted by `extractLeafSelector` (`shared/error-fingerprint.ts`). Pure helpers live in `shared/` (`locator-healing.ts`, `locator-fingerprint.ts`, `locator-healing.types.ts`); the reporter keeps a structural mirror (no `shared` import). Toggle with the `captureLocators` reporter option / `PIWI_CAPTURE_LOCATORS` env var (auto-off when `collectPerformanceMetrics` is false). Surfaced via `LocatorHealingPanel.vue` and the `locatorHealing` AI-diagnosis section. Re-run `npm run app:seed:demo` after changing the captured/stored shape.
- **Timed-out tests fold into `failedTests`**: There is no `timedOutTests` column on `test_runs`. Timed-out tests (per-case status `'timedOut'` from Playwright, camelCase; `'timedout'` lowercase in the declared `TestCaseStatus` union) are folded into `failedTests` so the run summary reconciles (`total = passed + failed + skipped + didNotRun`) and matches the UI (which already treats timed-out as failed in the status filter, color, and retry command). Every ingest site that writes `failedTests` MUST use `sumFailedAndTimedOut(body.failedTests, body.timedOutTests)` (for body-field sites: `finish.post.ts`, `submit.post.ts`, `upload.post.ts`, and the demo `app/demo/api/reporter.ts` mirror) or `countFailedFromTally(insertedStatusCounts)` (for per-status-tally sites: `events.post.ts` and the demo mirror). Helpers live in `shared/utils/test-counts.ts`. When adding a new ingest site, import and use these helpers — never write `failedTests: body.failedTests` directly. The reporter sends `timedOutTests` separately in the finish/submit body; `TestRunSubmitPayload`/`TestRunFinishPayload` declare it as optional.

## Environment
Expand Down
43 changes: 21 additions & 22 deletions application/app/components/shared/LocatorHealingPanel.vue
Original file line number Diff line number Diff line change
Expand Up @@ -6,22 +6,13 @@
*/

import { recommendLocatorFix } from '#shared/locator-healing';
import type { RankedLocator, LocatorFixRecommendation } from '#shared/locator-healing.types';
import type { RankedLocator, LocatorFixRecommendation, LocatorHealingResult } from '#shared/locator-healing.types';

const props = defineProps<{
runId: number;
testRunsCaseId: number;
}>();

interface LocatorHealingResult {
failingLocator: { method: string; args: Record<string, unknown> } | null;
fromPriorSuccess: RankedLocator[] | null;
fromElementMatch: RankedLocator[] | null;
fromAriaSnapshot: RankedLocator[] | null;
source: 'prior-run' | 'element-match' | 'fingerprint' | 'aria-snapshot' | 'none';
recommendation: LocatorFixRecommendation | null;
}

const {
data: healing,
pending,
Expand Down Expand Up @@ -64,18 +55,25 @@ const recommendationNote = computed(() => {
});

const sourceNote = computed(() => {
switch (healing.value?.source) {
case 'prior-run':
return 'Pre-captured from the last passing run — highest confidence';
case 'element-match':
return 'The element looks renamed or moved — these are fresh locators for its current identity on the failing page';
case 'fingerprint':
return 'Matched by locator signature (line numbers shifted)';
case 'aria-snapshot':
return 'Generated from the failure-time ARIA snapshot — limited, no HTML attributes';
default:
return '';
}
const note = (() => {
switch (healing.value?.source) {
case 'prior-run':
return 'Pre-captured from the last passing run — highest confidence';
case 'element-match':
return 'The element looks renamed or moved — these are fresh locators for its current identity on the failing page';
case 'fingerprint':
return 'Matched by locator signature (line numbers shifted)';
case 'cross-test':
return 'Captured by another test in this project that uses the same locator';
case 'aria-snapshot':
return 'Generated from the failure-time ARIA snapshot — limited, no HTML attributes';
default:
return '';
}
})();
// Stored snapshots age — surface when the data was last captured.
const captured = healing.value?.capturedAt;
return captured ? `${note} · captured ${formatRelativeTime(captured)}` : note;
});

// Subtitle color: green for pre-captured (high confidence), primary for a
Expand All @@ -84,6 +82,7 @@ const sourceClass = computed(() => {
switch (healing.value?.source) {
case 'prior-run':
case 'fingerprint':
case 'cross-test':
return 'text-success-600 dark:text-success-400';
case 'element-match':
return 'text-primary-600 dark:text-primary-400';
Expand Down
2 changes: 1 addition & 1 deletion application/app/utils/help-content.ts
Original file line number Diff line number Diff line change
Expand Up @@ -518,7 +518,7 @@ export const HELP_TOPICS = {
// ── Locator healing ────────────────────────────────────────────────────
'locator-healing': {
title: 'Alternative locators',
text: 'When a locator breaks after a UI change, Piwi suggests pre-captured alternatives from the last passing run. Each alternative is ranked by stability score — prefer data-testid (100) over CSS classes (10–40).',
text: 'When a locator breaks after a UI change, Piwi suggests pre-captured alternatives from the last passing run — or from another test in the project that uses the same locator. Each alternative is ranked by stability score — prefer data-testid (100) over CSS classes (10–40).',
doc: 'reporter#locator-healing',
},
} as const satisfies Record<string, HelpTopic>;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ defineRouteMeta({
tags: ['Test Runs'],
summary: 'Get locator healing suggestions for a failed test case',
description:
'Returns ranked alternative locator suggestions for a failing locator in a test run case. Uses pre-captured element snapshots from the last passing run, falling back to ARIA snapshot analysis.',
'Returns ranked alternative locator suggestions for a failing locator in a test run case. Uses pre-captured element snapshots from the last passing run (including snapshots captured by other tests in the project that use the same locator), falling back to ARIA snapshot analysis.',
parameters: [
{ name: 'id', in: 'path', required: true, schema: { type: 'integer' }, description: 'Test run id' },
{ name: 'caseId', in: 'path', required: true, schema: { type: 'integer' }, description: 'Test run case id' },
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
CREATE INDEX "idx_locator_snapshots_args_fp" ON "locator_snapshots" USING btree ("used_args_fp");
Loading