Skip to content

[BUG]: overall_score and the confusion matrix read different score sources, so identical objects can score 1.0 and FD at once #233

Description

@sromoam

Problem

Comparing two identical objects whose only field is an empty list produces a self-contradictory result:

class LineItemsInfo(StructuredModel):
    LineItemDays: Optional[List[str]] | Any = ComparableField(weight=1.0)
    match_threshold = 1.0

class Invoice(StructuredModel):
    LineItems: Optional[List[LineItemsInfo]] | Any = ComparableField(weight=1.0)

r = Invoice(LineItems=[LineItemsInfo(LineItemDays=[])]).compare_with(
    Invoice(LineItems=[LineItemsInfo(LineItemDays=[])]), include_confusion_matrix=True)

r["overall_score"]                                       # 1.0  -- a perfect match
r["confusion_matrix"]["fields"]["LineItems"]["overall"]   # tp=0, fd=1  -- a false discovery

The same pair is simultaneously a perfect match and a false discovery.

Root cause

Two different score sources for one pair, confirmed during the #225 review:

  • the confusion matrix reads the raw Hungarian similarity, which is 0.0 for an object whose only field is an empty list
  • overall_score reads the threshold-corrected score, which is 1.0

The underlying oddity is that an object with only an empty-list field gets list-path similarity 0.0 even when both sides are identical, so it lands below match_threshold and classifies as FD.

Scope

Pre-existing for multi-item lists: at n=2 both dev and the #225 branch give overall_score=1.0 with fd=2. #224 only extended it to the 1-vs-1 case as a consequence of making that case consistent, so this is not a regression from that PR.

Currently recorded only in a test docstring (tests/structured_object_evaluator/test_simple_list_in_structured_list.py::test_empty_simple_list_within_structured_list), which is why @vawsgit asked for it to be filed. That test deliberately asserts only the #224 property (fn == 0, fa == 0) and that the pair is counted exactly once, so it will not need rewriting when this is fixed -- but it does pin the contradiction's existence.

Proposed direction

Decide which is right and make both readers agree:

  1. Two identical objects should compare as 1.0, so the empty-list similarity is the bug and the pair should be a TP (or a list-level TN, since there is nothing to detect).
  2. Or the raw score is right and overall_score should not report 1.0.

Option 1 looks correct: identical inputs scoring 0.0 is hard to defend regardless of the downstream classification.

Acceptance criteria

  • overall_score and the confusion matrix cannot disagree about whether a pair matched
  • Two identical objects whose only field is an empty list are not a false discovery
  • test_empty_simple_list_within_structured_list is updated to pin the resolved behavior
  • Behavior verified at n=1 and n>1, since the contradiction exists at both

Notes

Surfaced by @vawsgit in the #225 review (finding 5) and re-raised in the re-review as worth filing rather than leaving in a docstring.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Status
    Backlog

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions