Skip to content

Test the JUnit dialects against their schemas, their own output and each other - #163

Merged
Paebbels merged 5 commits into
devfrom
claude/dialect-test-level
Aug 16, 2026
Merged

Test the JUnit dialects against their schemas, their own output and each other#163
Paebbels merged 5 commits into
devfrom
claude/dialect-test-level

Conversation

@eda-s-claude-bot

@eda-s-claude-bot eda-s-claude-bot Bot commented Aug 16, 2026

Copy link
Copy Markdown

New Features

  • tests/unit/JUnitDialects/ is a new test level for behaviour that only exists between a writer and a reader:
    • Schemas.py - every reference output in tests/data/JUnit validates against its dialect's schema, and the
      dialect's reader accepts what its schema accepts.
    • RoundTrip.py - per dialect: read a reference report, write it in the same dialect, validate the result and
      read it back, comparing test case count, test case names and hostnames.
    • Translation.py - the 5x5 conversion matrix: every dialect read and written as every dialect, each cell
      validated against the target schema and read back with the target reader.
  • tests/README.md writes down the testing strategy: the two phases (instantiation, combination), the four
    structural levels (package / module / class / method) and how they map onto this repository, why tests/data is
    evidence rather than fixtures, and the known-gap convention below.

Changes

  • Level (c) is one class per dialect, derived from a base class holding the checks; the base sets
    _dialectName = None and skips itself, so each check is written once and reported per dialect.

  • Format limits are asserted, not skipped. Translation.py::FORMAT_LIMITS names each pair with the data the
    target format needs and the source lacks, and the test asserts the conversion still fails:

    ("Any-JUnit", "pyTest-JUnit"):
        "pyTest-JUnit requires 'timestamp' on <testsuite>; the OSVVM report has none.",

    A skip goes quiet forever; this way lifting a limit turns the expectation red and the entry is removed by the
    pull-request that lifts it. A conversion failing for any other reason is a defect and does not belong in that
    table.

Known Issues

  • The six conversions in FORMAT_LIMITS fail correctly but not cleanly: Any-JUnitGoogleTest-JUnit
    raises AttributeError: 'NoneType' object has no attribute 'isoformat', and pyTest-JUnitCTest-JUnit
    writes a document that its own schema rejects rather than refusing. Fixed separately - a writer that cannot
    express a report should say so, not crash or emit invalid XML.
  • TestsuiteSummary.Aggregate() does not compute a start time from its children, which is why pyTest-JUnit
    CTest-JUnit is a limit at all: pytest's <testsuites> has no timestamp while its <testsuite> children do.
    Deriving it would turn two of the six limits into working conversions.

Unit Tests

tests/unit: 98 → 155 passed, 65 subtests, 11 skips (the base classes). The 5x5 matrix is 25 testcases.


Related Issues and Pull-Requests

  • Follows #162. Building this matrix is what found the last
    defect fixed there: Ant-JUnit4 and CTest-JUnit dropped the hostname of their root <testsuite> element.
  • The schema corrections these tests make safe - hostname as use="required" where every reference output has
    it, and Any-JUnit accepting a <testsuite> root - are deliberately left for a follow-up.

@eda-s-claude-bot
eda-s-claude-bot Bot requested a review from Paebbels as a code owner August 16, 2026 10:45
@codecov

codecov Bot commented Aug 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 31.72%. Comparing base (be86c31) to head (af97ec0).
⚠️ Report is 3 commits behind head on dev.

Additional details and impacted files
@@             Coverage Diff             @@
##              dev     #163       +/-   ##
===========================================
- Coverage   72.12%   31.72%   -40.41%     
===========================================
  Files           9        9               
  Lines        2490     2490               
  Branches      395      395               
===========================================
- Hits         1796      790     -1006     
- Misses        516     1700     +1184     
+ Partials      178        0      -178     
Flag Coverage Δ
unittests 31.72% <ø> (-40.41%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@eda-s-claude-bot eda-s-claude-bot Bot mentioned this pull request Aug 16, 2026
…ach other

The defects fixed in #162 all lived between a writer and a reader, and no test looked there: nothing validated
written output against the bundled schemas, and nothing converted one dialect into another. 24 test files and 94
testcases, zero schema assertions.

tests/unit/JUnitDialects/ adds that level:

* Schemas.py - every reference output in tests/data validates against its dialect's schema, and the reader accepts
  what the schema accepts. The reference outputs are ground truth: the schemas were reverse-engineered from them.
* RoundTrip.py - per dialect: read a reference report, write it in the same dialect, validate it, read it back, and
  compare test case count, test case names and hostnames.
* Translation.py - the 5x5 conversion matrix, each cell validated against the target schema and read back with the
  target dialect.

Where a conversion cannot work today, KNOWN_GAPS names the pair and the reason and the test asserts that it *still*
fails, so a fix turns the expectation red instead of passing unnoticed. Six pairs are listed; two of them are
format limits rather than defects. The same pattern records that Any-JUnit rejects a <testsuite>-rooted report
although it is the permissive dialect.

tests/README.md writes down the strategy: the two phases, the four structural levels and how they map onto this
repository, why tests/data is evidence rather than fixtures, and the known-gap convention.

tests/unit: 98 -> 155 passed, 65 subtests.

Co-Authored-By: Patrick Lehmann <Paebbels@gmail.com>
@Paebbels
Paebbels force-pushed the claude/dialect-test-level branch from d088f30 to e356eb0 Compare August 16, 2026 11:22
@Paebbels Paebbels added FileFormat: XML XML file format Unit Testing Unit test summaries labels Aug 16, 2026
Comment thread tests/unit/JUnitDialects/__init__.py Outdated
Comment thread tests/unit/JUnitDialects/__init__.py Outdated
Comment thread tests/unit/JUnitDialects/__init__.py
Comment thread tests/unit/JUnitDialects/RoundTrip.py Outdated
Comment thread tests/unit/JUnitDialects/Translation.py Outdated
* Dialect uses the ExtendedType metaclass with slots, declares its fields with type hints, and its read-only
  properties are marked @readonly - including the Dialect property of the three test modules.
* readReference() calls Aggregate() before handing the summary over, as the command line does before writing. Two
  of the four "defects" the matrix reported were this missing call: without it the writers see zero test cases and
  no duration, so they omitted 'time' and the target schema rejected the result.
* KNOWN_GAPS is FORMAT_LIMITS. Every remaining entry names data the target format requires and the source does not
  carry - a timestamp, or more than the one test suite the format holds. A conversion failing for any other reason
  is a defect and does not belong in that table.

Co-Authored-By: Patrick Lehmann <Paebbels@gmail.com>
Comment thread tests/unit/JUnitDialects/RoundTrip.py Outdated
Comment thread tests/README.md Outdated
claude-code and others added 2 commits August 16, 2026 17:13
The base classes derived from TestCase, so unittest collected them and pytest reported 11 skipped testcases that
test nothing. They are mixins now - SchemaMixin, RoundTripMixin, TranslationMixin - and each dialect class derives
from the mixin and TestCase.

They use 'metaclass=ExtendedType' without 'mixin=True': that option raises
'BaseClassWithoutSlotsError: Base-classes TestCase doesn't use __slots__', because unittest.TestCase has no
__slots__ and mixin=True requires them from every base class.

With the skip guard gone, tests/unit/JUnitDialects reports 57 passed and no skips.

Co-Authored-By: Patrick Lehmann <Paebbels@gmail.com>
unittest.TestCase is not created by ExtendedType and has no __slots__, so a mixin created by ExtendedType cannot
be combined with it. Classic mixins are the right tool there: plain classes, no metaclass.

The doc-strings and tests/README.md say why, so the next reader doesn't 'fix' them back.

Co-Authored-By: Patrick Lehmann <Paebbels@gmail.com>
Comment thread tests/unit/JUnitDialects/Schemas.py Outdated
@Paebbels
Paebbels merged commit 9e21158 into dev Aug 16, 2026
21 of 54 checks passed
@Paebbels
Paebbels deleted the claude/dialect-test-level branch August 16, 2026 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

FileFormat: XML XML file format Unit Testing Unit test summaries

Development

Successfully merging this pull request may close these issues.

2 participants