Description
The conformance package has two utilities for processing conformance result files:
conformance/src/unexpected_fails.py
conformance/src/validate_results.py
Neither currently has automated test coverage.
A coverage run of the conformance runner shows:
unexpected_fails.py: 0% coverage (22 statements)
validate_results.py: 0% coverage (62 statements)
Both utilities contain logic that would benefit from regression tests. unexpected_fails.py compares automated and manually recorded conformance results, while validate_results.py checks invariants in the generated TOML result files.
The existing result files currently pass both utilities:
unexpected_fails.py reports no discrepancies.
validate_results.py validates all 852 conformance result files without invariant violations.
Proposed change
Add focused automated tests covering the main behaviors of these utilities, including:
- matching automated and manual results
- detecting automated/manual result mismatches
- handling malformed TOML
- handling missing result keys
- accepting valid result files
- rejecting unknown result keys
- validating
conformance_automated values
- validating
conformant values
- handling missing
conformant when appropriate
- requiring non-empty
notes for partial conformance
This would provide regression coverage for the conformance result-processing utilities and make future changes to them safer.
Related: #1692
Description
The conformance package has two utilities for processing conformance result files:
conformance/src/unexpected_fails.pyconformance/src/validate_results.pyNeither currently has automated test coverage.
A coverage run of the conformance runner shows:
unexpected_fails.py: 0% coverage (22 statements)validate_results.py: 0% coverage (62 statements)Both utilities contain logic that would benefit from regression tests.
unexpected_fails.pycompares automated and manually recorded conformance results, whilevalidate_results.pychecks invariants in the generated TOML result files.The existing result files currently pass both utilities:
unexpected_fails.pyreports no discrepancies.validate_results.pyvalidates all 852 conformance result files without invariant violations.Proposed change
Add focused automated tests covering the main behaviors of these utilities, including:
conformance_automatedvaluesconformantvaluesconformantwhen appropriatenotesfor partial conformanceThis would provide regression coverage for the conformance result-processing utilities and make future changes to them safer.
Related: #1692