Add GPU test CI workflow with JIT and AOT jobs#1004
Merged
Conversation
achauveau-nv
force-pushed
the
achauveau/gpu-ci
branch
from
July 2, 2026 15:17
8826796 to
c885e31
Compare
matthewdcong
reviewed
Jul 9, 2026
| - &step-install-python-deps | ||
| name: Install Python deps | ||
| run: | | ||
| python3 -m venv /opt/venv |
Contributor
There was a problem hiding this comment.
We should use uv for virtual environment and package management instead of bare venv/pip. It's quite a bit faster and will save on CI usage.
Collaborator
Author
There was a problem hiding this comment.
In general I agree with the benefits of using this more modern tooling. However for the purpose of this PR I chose to stick with pip to remain consistent with current practice in the repo's existing workflows. Switching to uv is, I think, a worthy improvement to consider across all workflows, and separately from this PR.
rodlima78
approved these changes
Jul 21, 2026
achauveau-nv
force-pushed
the
achauveau/gpu-ci
branch
from
July 24, 2026 08:46
c885e31 to
eb792a3
Compare
Introduce .github/workflows/gpu_tests.yml, a test pipeline executing on self-hosted GPU runners. Two jobs run with CUDA 12.9: - jit-test-debug: editable install with DEBUG=1, JIT compiles CUDA extensions on first import; runs the full pytest suite. - aot-test-release: builds a wheel with 'pip wheel -v' (GSPLAT_DISABLE_JIT=1 triggers AOT compilation), installs it, then runs the pytest suite. Key implementation decisions: Jobs run in a Docker container and not directly on the runner to provide isolation. Dependencies install is performed during the workflow over a vanilla CUDA 12.9 container image: - It takes time and is not fully reproducible, but it makes a good starting point as the maintenance of custom container images would be heavier. - pip dependencies are cached into a persistent volume to improve performance. Both AOT and JIT compilation for gsplat and dependencies go through ccache, also with a persistent volume. Pulling already up-to-date compilation results from cache can offer several minutes of time saving per job. Container spec and five setup steps are shared via YAML anchors. GitHub's schema rejects unknown top-level keys, so there's no workflow-level node to hang them on; they're defined in the first job and reused by the second one. A root conftest.py marks a set of known-failing test_basic.py cases as xfail, gated on GPU_CI_XFAIL=1 set at the workflow level so the marker stays CI-only. These are marginal floating-point tolerance mismatches on the runner GPU rather than logic errors in the workflow setup and should be fixed as a follow-up. Trigger policy: the workflow runs automatically on push to main and is manually triggerable via workflow_dispatch (Actions UI or gh CLI). The pull_request trigger is intentionally omitted for now: job runtime is too long to gate every PR. The intent is to enable it by default once job execution time has been optimized.
Collaborator
Author
|
Auto-trigger on main PRs has been demoted at this stage after discussion with the team. The latest version was validated with an explicit trigger: https://github.com/nerfstudio-project/gsplat/actions/runs/30080994300 |
rodlima78
approved these changes
Jul 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce .github/workflows/gpu_tests.yml, a test pipeline executing on self-hosted GPU runners. Two jobs run with CUDA 12.9:
Key implementation decisions:
Jobs run in a Docker container and not directly on the runner to provide isolation.
Dependencies install is performed during the workflow over a vanilla CUDA 12.9 container image:
Both AOT and JIT compilation for gsplat and dependencies go through ccache, also with a persistent volume. Pulling already up-to-date compilation results from cache can offer several minutes of time saving per job.
Container spec and five setup steps are shared via YAML anchors. GitHub's schema rejects unknown top-level keys, so there's no workflow-level node to hang them on; they're defined in the first job and reused by the second one.
A root conftest.py marks a set of known-failing test_basic.py cases as xfail, gated on GPU_CI_XFAIL=1 set at the workflow level so the marker stays CI-only. These are marginal floating-point tolerance mismatches on the runner GPU rather than logic errors in the workflow setup and should be fixed as a follow-up.
Trigger policy: the workflow runs automatically on push to main and is manually triggerable via workflow_dispatch (Actions UI or gh CLI). The pull_request trigger is intentionally omitted for now: job runtime is too long to gate every PR. The intent is to enable it by default once job execution time has been optimized.