Problem
Comparing two identical objects whose only field is an empty list produces a self-contradictory result:
class LineItemsInfo(StructuredModel):
LineItemDays: Optional[List[str]] | Any = ComparableField(weight=1.0)
match_threshold = 1.0
class Invoice(StructuredModel):
LineItems: Optional[List[LineItemsInfo]] | Any = ComparableField(weight=1.0)
r = Invoice(LineItems=[LineItemsInfo(LineItemDays=[])]).compare_with(
Invoice(LineItems=[LineItemsInfo(LineItemDays=[])]), include_confusion_matrix=True)
r["overall_score"] # 1.0 -- a perfect match
r["confusion_matrix"]["fields"]["LineItems"]["overall"] # tp=0, fd=1 -- a false discovery
The same pair is simultaneously a perfect match and a false discovery.
Root cause
Two different score sources for one pair, confirmed during the #225 review:
- the confusion matrix reads the raw Hungarian similarity, which is
0.0 for an object whose only field is an empty list
overall_score reads the threshold-corrected score, which is 1.0
The underlying oddity is that an object with only an empty-list field gets list-path similarity 0.0 even when both sides are identical, so it lands below match_threshold and classifies as FD.
Scope
Pre-existing for multi-item lists: at n=2 both dev and the #225 branch give overall_score=1.0 with fd=2. #224 only extended it to the 1-vs-1 case as a consequence of making that case consistent, so this is not a regression from that PR.
Currently recorded only in a test docstring (tests/structured_object_evaluator/test_simple_list_in_structured_list.py::test_empty_simple_list_within_structured_list), which is why @vawsgit asked for it to be filed. That test deliberately asserts only the #224 property (fn == 0, fa == 0) and that the pair is counted exactly once, so it will not need rewriting when this is fixed -- but it does pin the contradiction's existence.
Proposed direction
Decide which is right and make both readers agree:
- Two identical objects should compare as
1.0, so the empty-list similarity is the bug and the pair should be a TP (or a list-level TN, since there is nothing to detect).
- Or the raw score is right and
overall_score should not report 1.0.
Option 1 looks correct: identical inputs scoring 0.0 is hard to defend regardless of the downstream classification.
Acceptance criteria
Notes
Surfaced by @vawsgit in the #225 review (finding 5) and re-raised in the re-review as worth filing rather than leaving in a docstring.
Problem
Comparing two identical objects whose only field is an empty list produces a self-contradictory result:
The same pair is simultaneously a perfect match and a false discovery.
Root cause
Two different score sources for one pair, confirmed during the #225 review:
0.0for an object whose only field is an empty listoverall_scorereads the threshold-corrected score, which is1.0The underlying oddity is that an object with only an empty-list field gets list-path similarity
0.0even when both sides are identical, so it lands belowmatch_thresholdand classifies as FD.Scope
Pre-existing for multi-item lists: at n=2 both
devand the #225 branch giveoverall_score=1.0withfd=2. #224 only extended it to the 1-vs-1 case as a consequence of making that case consistent, so this is not a regression from that PR.Currently recorded only in a test docstring (
tests/structured_object_evaluator/test_simple_list_in_structured_list.py::test_empty_simple_list_within_structured_list), which is why @vawsgit asked for it to be filed. That test deliberately asserts only the #224 property (fn == 0,fa == 0) and that the pair is counted exactly once, so it will not need rewriting when this is fixed -- but it does pin the contradiction's existence.Proposed direction
Decide which is right and make both readers agree:
1.0, so the empty-list similarity is the bug and the pair should be a TP (or a list-level TN, since there is nothing to detect).overall_scoreshould not report1.0.Option 1 looks correct: identical inputs scoring
0.0is hard to defend regardless of the downstream classification.Acceptance criteria
overall_scoreand the confusion matrix cannot disagree about whether a pair matchedtest_empty_simple_list_within_structured_listis updated to pin the resolved behaviorNotes
Surfaced by @vawsgit in the #225 review (finding 5) and re-raised in the re-review as worth filing rather than leaving in a docstring.