Testing harness

[dtinth]

Destination

Open-ended by design (dtinth): this map has no fixed endpoint — the direction is get the Jamulus codebase under test, and we chart more as we explore ("the mapping part of wayfinding is more useful than reaching a fixed destination"). Both unit tests and smoke tests are wanted eventually — different benefits, limitations, and confidence levels; separate PRs. Milestones along the way, not destinations:

1. First upstream PR: opened — jamulussoftware/jamulus#3828, in upstream review.
2. Smoke testing explored and eventually upstreamed (see its ticket).

Notes

• Tooling decided: qmake + QtTest, staying in-family with Jamulus.pro (no CMake/Catch2 for now).
• This fork is a free experimentation space; upstream PRs must stay very small.
• All upstream-facing activity (issues, PRs, review replies) is posted by @dtinth personally; the agent prepares code and draft text in this fork only.
• There are informal signals that upstream would welcome an automated testing harness.
• This effort carries execution: building the harness in the fork and preparing the upstream PR are in scope, not just decisions.
• Repo hygiene: no agent/tooling scratch files in the working tree.
• Skills for ticket sessions: /tdd (test implementation), /prototype (harness spikes), /grilling + /domain-modeling (decisions).
• Background: initial assessment and evidence + churn analysis in this issue's comments. Pre-map decisions recorded there: upstreaming is the goal, qmake+QtTest, first targets protocol.cpp → util → buffer.

Decisions so far

• Scope the first upstream PR — PR opened: jamulussoftware/jamulus#3828 (harness + 19-case protocol suite + 6-job CI, +1029 lines); scope/design settled through comment-by-comment review on this map; milestone 1 reached pending upstream review
• Split test helpers into a shared header — protocoltester.h precedent set; GenTestFrame retired for golden-hex snapshots (space-separated) + mutation of production frames
• Easy-to-read DSL: CProtocolTester fluent driver — kept; chains wrapped in QVERIFY2 + rich lastError() so failure messages identify the exact mismatch (anti-debugging-frenzy)
• Linux coverage job — gcovr HTML artifact + summary table; baseline: protocol.cpp 34.3% lines
• CI test reports — junitxml from every job + zero-permission step-summary rendering; checks-write action documented as opt-in alternative
• CI design for the test suite — 4-job matrix green in 26–66s (Linux Qt 5.15 + Qt 6, macOS Qt 6.10.2 w/ Xcode 16.3.0 pin, Windows MSVC), hard Totals: 41 passed gate on every platform; verified run 29926662716; triggers still need reshaping for upstream
• Prove the harness: QtTest seed suite builds and passes in the fork — proven: src/test/ harness links protocol.cpp+util.cpp with no stubs/production changes, 41/41 green incl. ASan/UBSan; dtinth: keep (structure review deferred), wire up CI on all three platforms next
• Upstream contribution requirements for a test/CI PR — anchor on still-open upstream Automatic Client/Server Smoke Test in CI jamulussoftware/jamulus#2428 (smoke-test spec, pre-agreed); AGPL 3.0+ short headers, Qt ≥ 5.12.2 + C++11, clang-format v14, test files under src/, one tiny workflow modeled on check-json-rpcs-docs.yml, target main; small CI PRs merge friction-free
• Track upstream #3819/#3821: GetValFromStream out-of-bounds guards — riskier to ride: fix semantics unsettled upstream; lead with ParseMessageFrame contract tests + a Add bounds check iChanID before array access jamulussoftware/jamulus#3810 regression test instead, cite GetValFromStream uses ASSERT which is likely compiled out during release builds jamulussoftware/jamulus#3819 only as context

Not yet specified

• Exact contents and narrative of the first upstream PR (sharpens after the prototype and research tickets resolve).
• How upstream review feedback gets worked once the PR is open.
• Later suites and their order: buffer/jitter regressions, wider util coverage, recorder golden-file tests, JSON-RPC surface tests.
• Smoke-test upstreaming (parked side-quest per dtinth, 2026-07-23): the headless build fixes (three one-liners — verification uncovered a third, link-stage bug the old workaround masked) are upstream as issue #3825 and PR #3826 (fronted by dtinth, first upstream PR of this effort). Once merged, the eventual smoke PR ships zero production changes.
• Fault-injection integration tier on top of the smoke test (TCP fallback per upstream PR Support TCP for protocol messages jamulussoftware/jamulus#3636 — pljones asked there for test cases): needs CAP_NET_ADMIN (netns/iptables/tc) so local/privileged-only, not per-PR. A third party (mcfnord) reports having such a netns harness and offered to share scripts — unverified, to be assessed when this graduates.
• Whether to revive fuzzing (afl-fuzz found the 2020 protocol bugs) on top of the harness, and what becomes of src/testbench.h.

Out of scope

• GUI and audio-backend (sound card) testing.

[dtinth-claw]

I surveyed the codebase (as of upstream main @ 49f4f8d) to see what "get it under test" would build on. Summary and a proposed starting plan below.

Current state: no automated tests at all

• Zero QtTest/QTest usage in Jamulus's own sources; no test directory, no test target, no CMake (qmake-only via Jamulus.pro). All *test* hits are vendored deps (libs/opus, libs/oboe).
• CI (autobuild.yml) is build/package/release only. The only quality gates are clang-format/shellcheck/pylint (coding-style-check.yml) and a JSON-RPC docs consistency check.
• The one in-tree artifact is src/testbench.h (CTestbench): a manual fuzz tool that fires random protocol messages at a UDP port on a 1 ms timer. Not wired into anything (one commented-out line in main.cpp), no assertions — but it proves CProtocol runs standalone.
• CONTRIBUTING.md's "Testing" section literally means "build it and try the feature."

What's in our favor

• Qt is already a dependency → QT += testlib (QtTest) costs nothing.
• The build already supports GUI-free compilation (headless, serveronly/SERVER_ONLY), so core code compiles without widgets.
• The best test targets are UI-free and deterministic:
  1. protocol.cpp (~3000 lines) — Create*Mes/Eval*Mes pairs are pure byte-buffer round-trips, and docs/JAMULUS_PROTOCOL.md is a spec to test against. Highest value: this is the wire format, and where malformed-input bugs live.
  2. buffer.h/.cpp — jitter/ring buffers (CNetBuf, CNetBufWithStats, CConvBuf), pure algorithmic.
  3. util.h/.cpp — CVector, CFIFO, CCRC, NetworkUtil, CHostAddress, etc.
  4. src/recorder/ — WAV + Reaper-project writers → golden-file tests.

Main friction: monolithic qmake project with heavy header coupling (global.h everywhere, SERVER_ONLY ifdefs), so a test target has to pull in a cluster of core .cpp files rather than link a tidy library.

Proposed first increment

1. tests/ with a qmake tests.pro (stay in-family with the existing build; no CMake migration) that compiles protocol.cpp, buffer.cpp, util.cpp + deps with SERVER_ONLY-ish config and links QtTest.
2. First suites: protocol message round-trip + malformed-input rejection (seeded by CTestbench's generators, but with assertions), CNetBuf put/get/overflow behavior, CCRC/NetworkUtil basics.
3. A small test.yml CI job (ubuntu, Qt from apt) running the suite on PRs — separate from autobuild.yml so it stays fast.

Open questions

1. Destination: is this fork-only infrastructure, or do we shape it for eventual upstreaming to jamulussoftware/jamulus? Upstream maintainers are currently wary of high AI-generated volume, so if upstreaming is the goal I'd keep the first PR very small (harness + one suite) and discuss with them first.
2. OK with qmake + QtTest (my recommendation, least invasive), or would you rather introduce CMake/Catch2 for the test side?

@dtinth — blocked on question 1 (and 2 if you have a preference); happy to start on the harness + protocol round-trip suite on a branch here in the meantime.

[dtinth]

"and where malformed-input bugs live"

Anything to back this up? Past issues, commits? Can you also check the issue tracker and past work on what has a large amount of churn, that would be a good first target?

Re. questions:

1. Yes, planning on upstreaming. Agree that first PR should be very small, but in our fork we can experiment as much as we want, we can decide the order of upstreaming later.
2. As you recommended.

Would /wayfinder be a good fit for this? If /setup-matt-pocock-skills is needed, please remember information in local memory so it (1) persists between Git branches (2) keeps the Git repo clean of agent files (for now).

[dtinth-claw]

Evidence for "malformed-input bugs live in protocol parsing"

Yes, this class has a documented track record here — and one member of it is open right now:

• GetValFromStream uses ASSERT which is likely compiled out during release builds jamulussoftware/jamulus#3819 + PR Guard GetValFromStream against out-of-bounds reads in release builds jamulussoftware/jamulus#3821 (opened this week, open) — GetValFromStream in protocol.cpp validates its arguments only with Q_ASSERT, which compiles out in release builds; CVector uses std::vector's unchecked operator[], so a truncated/malformed network message can read past the end of the buffer in the builds people actually ship.
• Check the size of PROTMESSID_ACKN messages jamulussoftware/jamulus#302 → commit 024ebb47 (2020) — an ACKN message with a valid checksum but no payload caused an out-of-bounds read (every other Eval* checked size first; the ACKN special case didn't). Commit message: "Found by fuzzing with afl-fuzz."
• Fix two uninitialised reads jamulussoftware/jamulus#382 (2020) — "Fix two uninitialised reads. Found using valgrind with fuzzer-generated testcases."
• jamulus 3.12.1 headless crash with out-of-bounds vector access jamulussoftware/jamulus#3747 → commit d545dbce (3.12.1, this year) — headless server crash from an out-of-bounds vector access in buffer.cpp.
• Add bounds check iChanID before array access jamulussoftware/jamulus#3810 (merged last week) — bounds-check a network-provided iChanID before array indexing.

So both the 2020 fuzzing wave and the 2026 bugs are the same shape: network-derived values used as sizes/indices without release-mode guards. None of those fixes came with a regression test, because there's nowhere to put one — which is precisely this issue.

Also relevant precedent: upstream's own jamulussoftware#1413 "Run tests as part of build/CI" (2021) proposed running testbench.h in CI; it produced a useful explanation from atsampson (it's a template fuzzer generating valid-but-random messages) but was closed in 2022 without any tests landing.

Churn analysis (commits touching each file since 2021-07, excluding sweeps that touched >10 files)

file	commits	unit-testable?
util.cpp / util.h	57 / 47	yes (pure-logic half)
main.cpp	42	no (entry point)
clientdlg.cpp / serverdlg.cpp	40 / 39	no (GUI)
server.cpp / client.cpp	34 / 34	partially (orchestration)
socket.cpp	31	parse path yes
serverlist.cpp	26	mostly
settings.cpp	21	yes (touches QSettings)
protocol.cpp	11	yes, ideal

Interesting split: churn and bug-severity point at different files. util.cpp/h is the churn champion (it's a grab-bag everyone touches), while protocol.cpp is low-churn but is where the security-class bugs recur — low churn there also means tests won't rot.

Proposed first target

1. protocol.cpp malformed-input + round-trip suite — every Create*/Eval* pair round-trips, and truncated/corrupted variants must be rejected without OOB access. This directly gives GetValFromStream uses ASSERT which is likely compiled out during release builds jamulussoftware/jamulus#3819/Guard GetValFromStream against out-of-bounds reads in release builds jamulussoftware/jamulus#3821 the regression test it currently can't have, which also makes a great small first upstream PR (fix + guard + test + harness in one story).
2. util.h data-structure suite (CVector, CFIFO, CCRC, NetworkUtil) — covers the churn champion.
3. buffer.cpp jitter-buffer suite (regression for jamulus 3.12.1 headless crash with out-of-bounds vector access jamulussoftware/jamulus#3747's class) as the third.

Next step on my side: harness branch in this fork (qmake + QtTest as agreed) with target 1 as the seed suite.

Re /wayfinder / /setup-matt-pocock-skills: neither is installed in my current environment, so I can't evaluate wayfinder's fit from here — if you install it on this box I'll pick it up. Your constraints are saved to local memory either way: agent/skill scratch files stay out of the git repo, and cross-branch state lives in local memory, not the working tree.

[dtinth]

I updated the skills in the dev box, please check.