Skip to content

Add automated tests for conformance result utilities #2337

Description

@BHUVANSH855

Description

The conformance package has two utilities for processing conformance result files:

  • conformance/src/unexpected_fails.py
  • conformance/src/validate_results.py

Neither currently has automated test coverage.

A coverage run of the conformance runner shows:

  • unexpected_fails.py: 0% coverage (22 statements)
  • validate_results.py: 0% coverage (62 statements)

Both utilities contain logic that would benefit from regression tests. unexpected_fails.py compares automated and manually recorded conformance results, while validate_results.py checks invariants in the generated TOML result files.

The existing result files currently pass both utilities:

  • unexpected_fails.py reports no discrepancies.
  • validate_results.py validates all 852 conformance result files without invariant violations.

Proposed change

Add focused automated tests covering the main behaviors of these utilities, including:

  • matching automated and manual results
  • detecting automated/manual result mismatches
  • handling malformed TOML
  • handling missing result keys
  • accepting valid result files
  • rejecting unknown result keys
  • validating conformance_automated values
  • validating conformant values
  • handling missing conformant when appropriate
  • requiring non-empty notes for partial conformance

This would provide regression coverage for the conformance result-processing utilities and make future changes to them safer.

Related: #1692

Metadata

Metadata

Assignees

No one assigned

    Labels

    topic: otherOther topics not covered

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions