Test the JUnit dialects against their schemas, their own output and each other - #163
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #163 +/- ##
===========================================
- Coverage 72.12% 31.72% -40.41%
===========================================
Files 9 9
Lines 2490 2490
Branches 395 395
===========================================
- Hits 1796 790 -1006
- Misses 516 1700 +1184
+ Partials 178 0 -178
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
Open
…ach other The defects fixed in #162 all lived between a writer and a reader, and no test looked there: nothing validated written output against the bundled schemas, and nothing converted one dialect into another. 24 test files and 94 testcases, zero schema assertions. tests/unit/JUnitDialects/ adds that level: * Schemas.py - every reference output in tests/data validates against its dialect's schema, and the reader accepts what the schema accepts. The reference outputs are ground truth: the schemas were reverse-engineered from them. * RoundTrip.py - per dialect: read a reference report, write it in the same dialect, validate it, read it back, and compare test case count, test case names and hostnames. * Translation.py - the 5x5 conversion matrix, each cell validated against the target schema and read back with the target dialect. Where a conversion cannot work today, KNOWN_GAPS names the pair and the reason and the test asserts that it *still* fails, so a fix turns the expectation red instead of passing unnoticed. Six pairs are listed; two of them are format limits rather than defects. The same pattern records that Any-JUnit rejects a <testsuite>-rooted report although it is the permissive dialect. tests/README.md writes down the strategy: the two phases, the four structural levels and how they map onto this repository, why tests/data is evidence rather than fixtures, and the known-gap convention. tests/unit: 98 -> 155 passed, 65 subtests. Co-Authored-By: Patrick Lehmann <Paebbels@gmail.com>
Paebbels
force-pushed
the
claude/dialect-test-level
branch
from
August 16, 2026 11:22
d088f30 to
e356eb0
Compare
Paebbels
reviewed
Aug 16, 2026
* Dialect uses the ExtendedType metaclass with slots, declares its fields with type hints, and its read-only properties are marked @readonly - including the Dialect property of the three test modules. * readReference() calls Aggregate() before handing the summary over, as the command line does before writing. Two of the four "defects" the matrix reported were this missing call: without it the writers see zero test cases and no duration, so they omitted 'time' and the target schema rejected the result. * KNOWN_GAPS is FORMAT_LIMITS. Every remaining entry names data the target format requires and the source does not carry - a timestamp, or more than the one test suite the format holds. A conversion failing for any other reason is a defect and does not belong in that table. Co-Authored-By: Patrick Lehmann <Paebbels@gmail.com>
Paebbels
reviewed
Aug 16, 2026
The base classes derived from TestCase, so unittest collected them and pytest reported 11 skipped testcases that test nothing. They are mixins now - SchemaMixin, RoundTripMixin, TranslationMixin - and each dialect class derives from the mixin and TestCase. They use 'metaclass=ExtendedType' without 'mixin=True': that option raises 'BaseClassWithoutSlotsError: Base-classes TestCase doesn't use __slots__', because unittest.TestCase has no __slots__ and mixin=True requires them from every base class. With the skip guard gone, tests/unit/JUnitDialects reports 57 passed and no skips. Co-Authored-By: Patrick Lehmann <Paebbels@gmail.com>
unittest.TestCase is not created by ExtendedType and has no __slots__, so a mixin created by ExtendedType cannot be combined with it. Classic mixins are the right tool there: plain classes, no metaclass. The doc-strings and tests/README.md say why, so the next reader doesn't 'fix' them back. Co-Authored-By: Patrick Lehmann <Paebbels@gmail.com>
Paebbels
reviewed
Aug 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New Features
tests/unit/JUnitDialects/is a new test level for behaviour that only exists between a writer and a reader:Schemas.py- every reference output intests/data/JUnitvalidates against its dialect's schema, and thedialect's reader accepts what its schema accepts.
RoundTrip.py- per dialect: read a reference report, write it in the same dialect, validate the result andread it back, comparing test case count, test case names and hostnames.
Translation.py- the 5x5 conversion matrix: every dialect read and written as every dialect, each cellvalidated against the target schema and read back with the target reader.
tests/README.mdwrites down the testing strategy: the two phases (instantiation, combination), the fourstructural levels (package / module / class / method) and how they map onto this repository, why
tests/dataisevidence rather than fixtures, and the known-gap convention below.
Changes
Level (c) is one class per dialect, derived from a base class holding the checks; the base sets
_dialectName = Noneand skips itself, so each check is written once and reported per dialect.Format limits are asserted, not skipped.
Translation.py::FORMAT_LIMITSnames each pair with the data thetarget format needs and the source lacks, and the test asserts the conversion still fails:
A skip goes quiet forever; this way lifting a limit turns the expectation red and the entry is removed by the
pull-request that lifts it. A conversion failing for any other reason is a defect and does not belong in that
table.
Known Issues
FORMAT_LIMITSfail correctly but not cleanly:Any-JUnit→GoogleTest-JUnitraises
AttributeError: 'NoneType' object has no attribute 'isoformat', andpyTest-JUnit→CTest-JUnitwrites a document that its own schema rejects rather than refusing. Fixed separately - a writer that cannot
express a report should say so, not crash or emit invalid XML.
TestsuiteSummary.Aggregate()does not compute a start time from its children, which is whypyTest-JUnit→CTest-JUnitis a limit at all: pytest's<testsuites>has no timestamp while its<testsuite>children do.Deriving it would turn two of the six limits into working conversions.
Unit Tests
tests/unit: 98 → 155 passed, 65 subtests, 11 skips (the base classes). The 5x5 matrix is 25 testcases.Related Issues and Pull-Requests
defect fixed there:
Ant-JUnit4andCTest-JUnitdropped the hostname of their root<testsuite>element.hostnameasuse="required"where every reference output hasit, and
Any-JUnitaccepting a<testsuite>root - are deliberately left for a follow-up.