A parallel, self-balancing CI pipeline for FINN [CI 8/9]

[merkelmarrow]

This is PR 8 of 9 of a series intended to make CI faster and more robust.

This PR overhauls FINN's main Jenkins build job with a parallel build pipeline. Instead of one long serial run, the test matrix is split into shards that run side by side across the available build agents, and the pipeline learns from past runs to keep those shards balanced over time.

The pipeline itself delegates as much logic as possible to Python. All of the real logic (such as which tests run, how they are sharded, how long they took last time, what to keep on disk, how to clean up) lives in the dependency-free finn_ci Python package that can be unit tested easily. The Jenkins side calls into it frequently. This is the main strategy shift.

Why this matters

• Much faster CI. Work that used to run in sequence now runs in parallel, cutting CI times by about 75%.
• Self-balancing. The pipeline records how long each group of tests takes, and uses that history to spread work evenly across shards automatically.
• Reliability and observability. The diff is very large. However, a very large proportion of this is mechanical error handling and error observability, which is the difference between a CI pipeline that can be repaired easily and a flaky one that will be abandoned. These robustness measures were developed and refined in response to real failure modes observed during over 80 Jenkins runs over the span of 3 months.
• Testability. CI logic now mainly lives in testable Python, and configuration drift between the pipeline definition, the docs, and the Python source of truth is caught by a unit test.

Using it

For most contributors, nothing changes. You still write tests with the same markers, and you can run exactly what CI runs locally with a regular pytest invocation through Docker. There is a README guide for contributors and maintainers included with this change.

For Jenkins operators, a single shared-storage directory is the only additional setting needed. With it set, the pipeline gets the shared image cache, persistent timing history, per-agent download caches and hardware bitstream handoff. Retention is handled automatically, with old images, build artifacts and timing snapshots pruned according to the pruning policy in finn_ci.retention.

This change also adds some optional helpers for LSF integration.

Behaviour

All logic has been tried and tested in a real lab environment. The failure paths learned from shared NFS, flaky build agents, orphaned LSF jobs, aborted builds etc. are much of the value of the pipeline. On failure, it surfaces the relevant tool logs and per-test failure summaries directly in the build output, and fails fast with clear errors when something can be caught at the validation stage.

The pipeline is now much more resilient and flexible to unknown variables: it can safely delegate tool calls to a HPC cluster, or run everything locally, adapt to any number of build machines, use networked storage or fall back to local scratch.

It also lays the groundwork for the HW Jenkinsfile. Successful builds aggregate one package per board with a readiness marker that a later hardware pipeline will pick up (PR 9 of this series).

Breaking changes

As it stands, Jenkinsfile_HW has not been moved or changed. This is so that it can continue running with any saved artifacts in the legacy ARTIFACT_DIR until the migration is complete. The location of the CI, Brevitas, and full Jenkinsfiles will need to be updated in the relevant DSL jobs on the deployment side (they were moved to the ci/ subdirectory). This is a breaking change for anyone running Jenkins with FINN. Otherwise, there are no other breaking changes, and only added functionality by configuring a shared storage location (FINN_CI_NFS_ROOT) on your deployment.

Test/review

I recommend reading the included README during review. The finn_ci package is unit tested without a FINN install. To run the tests locally:

python -m pytest tests/util/test_finn_ci_*.py tests/util/test_ci_config_sync.py