fix(whois): Jaccard org similarity to stop containment over-corroborating (ENG-5172)

[josephwhenry]

Fixes ENG-5172. Supersedes #122 and #127 — see Why a new PR below.

The bug, on main today

whois.Corroborate scores org names with strutil.TokenSimilarity, which divides by the shorter token set (containment). Any resolved registrant that merely contains a single-token pivot org scores 1.0 and reports "match":

Acme         vs Acme Enterprises LLC                       sim=1.0000  -> match
Okta         vs Okta Security Inc.                         sim=1.0000  -> match
Walmart      vs Walmart Global Enterprises Holdings LLC    sim=1.0000  -> match
Praetorian   vs Praetorian Inc.                            sim=1.0000  -> match   (correct)

This is ENG-5172 unchanged. The WHOIS consolidation deleted reverse_whois_verify.go, but decideConfidence was reincarnated as whois.Corroborate with the same 0.60/0.30 thresholds and the same containment metric, so the defect survived the refactor intact.

Blast radius today is small: Corroboration is a metadata string on the finding (whoisFindingData.Corroboration, carried in Guard at p_whois.go) and does not currently feed confidence scoring. That makes now the cheap moment to correct the metric, before it becomes load-bearing.

Change

1. Jaccard instead of containment. OrgSimilarity now scores strutil.JaccardTokenSets — shared distinct tokens ÷ union — so both sides' distinguishing tokens count against the score. Same output on the same inputs:

Acme         vs Acme Enterprises LLC                       sim=0.5000  -> unverifiable
Okta         vs Okta Security Inc.                         sim=0.5000  -> unverifiable
Walmart      vs Walmart Global Enterprises Holdings LLC    sim=0.2500  -> unverifiable
Praetorian   vs Praetorian Inc.                            sim=1.0000  -> match

Genuine equality after legal-suffix stripping (Acme vs Acme Corp → {acme} vs {acme}) still scores 1.0 and still matches.

2. A containment guard on the mismatch arm — the other half of the fix. Jaccard alone over-corrects in the opposite direction. Because it divides by the union, a contained pivot sinks on length asymmetry alone: Walmart above lands at 0.25, below the 0.30 floor, so a bare threshold would report a plausible subsidiary as a contradiction — a strictly worse verdict than an unresolved lookup. strutil.TokenSetContained gates the arm: containment in either direction (the pivot may be the longer name) is under-specification, not disagreement, and falls through to "unverifiable". Reporting "mismatch" now requires each side to contribute a token the other lacks.

Note in the output above that Walmart sits below the mismatch floor and still correctly reports unverifiable — that is the guard working. Do not take Jaccard without it.

Thresholds are unchanged at 0.60 / 0.30, now Jaccard-interpreted. Mismatch stays strictly-less-than. An org that reduces to zero tokens after suffix stripping ("Co., Ltd.") is "unverifiable" — similarity is undefined, not zero.

3. github_org deliberately keeps containment. It calls strutil.TokenSimilarity and uses the result as a weight (0.25 * similarity) on a weak hint, not as a threshold gate — a display name containing the target org name is exactly the signal it wants to reward at full weight, and Jaccard would penalize it for length asymmetry. The two callers score differently on purpose, and the reason is now recorded at the call site instead of being an accident of where the old helper happened to live. Switching github_org onto OrgSimilarity is defensible on its merits (suffix stripping is better for org names) but it is a scoring change to a shipped plugin with its own test updates, so it does not belong in a bug fix.

4. CLAUDE.md. Replaces the Reverse-WHOIS Verify-After-Retrieve section, which documented reverse_whois_verify.go, verifyCandidates, decideConfidence, registrantResolver and whoisclient.go — all deleted by the consolidation. It was stale on main independent of this fix. The referral-hardening bullet describes surviving code and is retained with its current symbol names (tcp43RawDial, netutil.SSRFSafeControl, maxResponseBytes, maxReferrals).

Placement

The primitives go in pkg/lib/strutil beside TokenSimilarity; the org-aware wrapper stays in pkg/whois. Both take token slices so Corroborate tokenizes once and feeds the same tokens to the metric and the guard. normalizeOrg becomes normalizeOrgTokens for that reason.

Scope is contained: only Corroborate called OrgSimilarity, only whois.go calls Corroborate, and there are no callers in guard, aurelian, or praetorian-cli.

Why a new PR instead of resolving #122 / #127

git merge-tree for #122 against main:

CONFLICT (content):       pkg/plugins/domains/github_org.go
CONFLICT (modify/delete): reverse_whois_verify.go deleted in main
CONFLICT (modify/delete): reverse_whois_verify_test.go deleted in main

The correct resolution for the modify/delete pairs is git rm both — which leaves #122 holding one uncalled function plus edits to prose about deleted code. That is a reimplementation, not a conflict resolution. #127 compounds it: it is stacked on #122's branch, and by its own description carries zero production lines (its deliverable is ~52 lines of comment in the deleted file plus a calibration test). The one argument for keeping them split — #122 the fix, #127 the calibration record — collapses when the fix is ~5 lines of production code.

#122's jaccardTokenSets and tokenSetContained are carried forward here essentially verbatim, along with their test tables and the normalizeOrgTokens refactor. #127's reachability enumeration is deliberately not ported: it defends thresholds that are not currently load-bearing, at ~1300 lines. It can be revisited if/when Corroboration starts driving scores.

#129 has no analogue here. It patches summarizeVerifyPass over whoisIncompleteness — the entire incompleteness-tracking subsystem was deleted, and nothing in pkg/whois replaces it. It should be closed outright.

Tests

• pkg/lib/strutil/strutil_test.go (new) — TestJaccardTokenSets pins the metric including the containment case; TestJaccardTokenSets_PenalizesContainmentUnlikeTokenSimilarity asserts the contrast between the two metrics directly, so the reason both exist survives as an assertion rather than a comment; TestTokenSetContained covers direction symmetry, equal sets, disjoint sets, private-tokens-both-sides, empty sides, and duplicate collapse.
• TestCorroborate gains 8 cases: both over-corroboration shapes, both containment directions for the over-correction, the single-token exact match that must keep working, a sparse non-contained pair that must still mismatch (so the arm is proven reachable through the guard), and the all-suffix org.
• TestOrgSimilarity gains the containment values that previously returned 1.0, plus argument symmetry.
• All four pre-existing TestOrgSimilarity assertions and all eight pre-existing TestCorroborate cases pass unchanged — they were all equality-after-stripping cases, which is precisely why the table never caught this.

Verification

go build ./cmd/pius ok · go vet ./... clean · go test -count=1 ./... green, 0 failures · gofmt -l output is byte-identical to main's (7 files of pre-existing drift; this branch adds none, and gofmt -d on github_org.go shows only a pre-existing trailing blank line). All tests hermetic — no network, no keys.

🤖 Generated with Claude Code

[coderabbitai]

Walkthrough

The change adds Jaccard token-set similarity and symmetric containment detection. WHOIS organization normalization now removes legal suffixes and compares distinct token sets. Corroboration returns unverifiable for empty or contained sets, and reports mismatches only for low-similarity, non-contained sets. Tests cover scoring, containment, normalization, and corroboration outcomes. Documentation records the updated workflow and WHOIS referral limits.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch johnnovak/eng-5172-jaccard-org-similarity

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

pkg/whois/normalize_test.go (1)

94-96: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Correct the suffix-only test description and add the reversed case.

"Acme Corp" normalizes to {acme}. It does not reduce to zero tokens. State that the pivot reduces to zero tokens. Add a case where the resolved organization is "Co., Ltd." to preserve symmetric coverage.

Proposed update

-		// Both sides reduce to nothing after legal-suffix stripping, so similarity
+		// The pivot reduces to nothing after legal-suffix stripping, so similarity
 		// is undefined — unverifiable, never a mismatch.
 		{"all-suffix orgs are unverifiable", "Co., Ltd.", "Acme Corp", "unverifiable"},
+		{"all-suffix resolved org is unverifiable", "Acme Corp", "Co., Ltd.", "unverifiable"},

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/whois/normalize_test.go` around lines 94 - 96, Update the test case
description in the similarity table to state that the pivot reduces to zero
tokens, not both organizations. Add the reversed case with resolved organization
“Co., Ltd.” and the corresponding pivot organization so suffix-only
normalization remains covered symmetrically.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@pkg/whois/normalize_test.go`:
- Around line 94-96: Update the test case description in the similarity table to
state that the pivot reduces to zero tokens, not both organizations. Add the
reversed case with resolved organization “Co., Ltd.” and the corresponding pivot
organization so suffix-only normalization remains covered symmetrically.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 08992688-67d8-4655-a1c5-224299a599be

📥 Commits

Reviewing files that changed from the base of the PR and between 823dcd2 and d9b6591.

📒 Files selected for processing (6)

• CLAUDE.md
• pkg/lib/strutil/strutil.go
• pkg/lib/strutil/strutil_test.go
• pkg/plugins/domains/github_org.go
• pkg/whois/normalize.go
• pkg/whois/normalize_test.go

[github-actions]

Codex Review

Critical Issues

None.

Security

No security concerns flagged.

Suggestions

No suggestions.

Verification: attempted go test ./pkg/lib/strutil ./pkg/whois ./pkg/plugins/domains, but it failed before running because the read-only filesystem could not create /home/runner/go.

---

Reviewed by Codex (gpt-5.5)

[github-actions]

Claude Review

Critical issues

None. The fix is correct and well-scoped. Corroborate guards zero-token sides before the Jaccard division (safe), the mismatch arm is gated by TokenSetContained in both directions, and the empty-pivot ("") / privacy (unverifiable) short-circuits are preserved above the changed hunk. normalizeOrg -> normalizeOrgTokens drops the only strings.Join use but strings stays heavily used elsewhere in the file, so no compile break. Keeping github_org on containment (a weight, not a gate) is the right call and is now documented at the call site.

Security

No security concerns flagged. Pure scoring-metric change: no new network calls, auth, or input-trust surface; the SSRF / byte-cap / hop-cap hardening described in CLAUDE.md is pre-existing.

Test coverage

Present and thorough. New strutil_test.go pins the Jaccard-vs-containment contrast, direction symmetry, and duplicate collapse; TestCorroborate / TestOrgSimilarity add both over-corroboration shapes, both containment directions, the single-token exact match, a reachable non-contained mismatch, and the all-suffix case.

No critical issues — LGTM pending human review.

[github-actions]

Gemini Review

Critical Issues

None.

Security

No security concerns flagged.

Suggestions

• TokenSimilarity score inflation: In pkg/lib/strutil/strutil.go, TokenSimilarity currently iterates over the raw shorter slice rather than its distinct token set. If a string contains repeated tokens (e.g., "acme acme") and the other string contains just one instance of it ("acme widgets"), matches is incremented for every duplicate. This allows the metric to artificially inflate to 1.0. Consider adopting the distinct map approach for shorter to prevent this edge case.
• Unused exported function: OrgSimilarity in pkg/whois/normalize.go is no longer used internally, as Corroborate was refactored to inline the tokenization step and call strutil.JaccardTokenSets directly (avoiding duplicate work). If this function is not part of a public API consumed by other repositories, it and its associated tests can be safely removed as dead code.

---

Reviewed by Gemini (gemini-3.1-pro-preview)

[josephwhenry]

Closing — reopened from the correctly-named branch. This PR was raised from johnnovak/eng-5172-jaccard-org-similarity, a branch name I chose in error: I took the johnnovak/ prefix from #122 because it shares the ENG-5172 ticket, but the convention here is author namespacing, and the work is Joseph's (the commit was authored as josephwhenry throughout). GitHub cannot retarget an open PR's head branch, so this is closed in favor of a new PR from jwh/consolidate-similarities. Identical commit, no content change.