[FEATURE]: a comparator for normalized text equality

[sromoam]

Problem

Before 0.7.0, ExactComparator quietly lowercased inputs and stripped punctuation and whitespace before comparing. #199 established that this makes it not exact, and #220 reduced it to a single case_sensitive flag, removing the punctuation and whitespace handling.

That capability is legitimately useful and now has no home. Comparing "U.S.A." to "USA", or "John Smith" to "John Smith", is a real evaluation need — for OCR-derived text and LLM output especially, where formatting drift is noise rather than signal. Today the closest approximation is LevenshteinComparator(threshold=1.0), which is neither obvious nor exactly right.

Why it isn't just a flag on ExactComparator

Worth recording, because it's the reason this needs its own comparator rather than a boolean: there is no agreed definition of "strip punctuation." Three plausible definitions disagree on 9 of 16 realistic inputs:

input	keep alphanumerics	string.punctuation (what we used)	Unicode P*
U.S.A.	USA	USA	USA
a—b (em dash)	ab	a—b	ab
it's (curly quote)	its	it's	its
$1,247.50	124750	124750	$124750
x±y	xy	x±y	x±y
emoji🎉here	emojihere	emoji🎉here	emoji🎉here

$, ±, and emoji are Unicode Symbols, not Punctuation, so a P* strip keeps them. The inverse framing ("ignore anything that isn't a character") has its own trap: on côte written with a combining acute accent, keeping only Letters and Numbers deletes the accent, turning côte (coast) into cote (quota).

So the choice is genuinely data-dependent, and whatever this comparator does should be stated rather than assumed.

There's also a latent bug to fix here

Our existing strip_punctuation_space uses string.punctuation — 32 ASCII characters — so em dashes, curly quotes, non-breaking spaces, and full-width punctuation pass through unnormalized. Any LLM-derived text hits this. Nobody has reported it because it fails silently in the lenient direction, inflating scores. Whatever lands here should not inherit that limitation.

What success looks like

• A user who wants punctuation- or whitespace-insensitive equality has an obvious comparator to reach for, and can tell from its documentation exactly what it will and won't ignore.
• Anyone who relied on pre-0.7.0 ExactComparator has a documented way to get that behavior back.
• It works with the rest of the machinery: registry, to_json_schema() / from_json_schema() round-trip, and explain().
• Unicode text is handled correctly, including the accent case above.

Deliberately not specifying the API here. Whether it's a set of transform flags, a few named presets, a normalizer pipeline, or something else is the interesting part of the work, and the person building it will have better information than this ticket does. Same for the name — NormalizedComparator fits our existing vocabulary and one-word-intent convention, but it's not settled.

Notes

Came out of #199 and the #220 review, where we decided ExactComparator should mean == plus at most Unicode case folding — the one transform with a real standard behind it — and that everything ambiguous belongs in a comparator that can be explicit about its choices. Related: #199, #220.