chokevin · chokevin · May 5, 2026 · May 5, 2026
diff --git a/.github/workflows/build-swordfish-image.yml b/.github/workflows/build-swordfish-image.yml
@@ -4,9 +4,12 @@ name: build-swordfish-image
 # build context. Tags: branch name, short SHA, latest on main, optional
 # version tag via workflow_dispatch.
 #
-# Note: the base is nvcr.io/nvidia/pytorch:25.03-py3 (~12GB), so this build
-# is slow (~15-30min on ubuntu-latest). We reclaim ~30GB of preinstalled
-# tooling first to avoid running out of disk on the base image pull.
+# Note: GitHub Actions builds use the public nvcr.io/nvidia/pytorch:25.03-py3
+# base (~12GB), so this build is slow (~15-30min on ubuntu-latest). The
+# Dockerfile default remains the private in-region ACR base used by the
+# canonical Azure ACR build path; Actions cannot pull that base without Azure
+# credentials. We reclaim ~30GB of preinstalled tooling first to avoid running
+# out of disk on the public base image pull.
 
 on:
   push:
@@ -94,6 +97,7 @@ jobs:
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
           build-args: |
+            BASE_IMAGE=nvcr.io/nvidia/pytorch:25.03-py3
             ${{ inputs.liger_version && format('LIGER_VERSION={0}', inputs.liger_version) || '' }}
             ${{ inputs.liger_ref && format('LIGER_REF={0}', inputs.liger_ref) || '' }}
             SWORDFISH_SHA=${{ github.sha }}

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -24,5 +24,5 @@ jobs:
   #   runs-on: [self-hosted, a100]
   #   steps:
   #     - uses: actions/checkout@v4
-  #     - run: pip install uv && uv sync
+  #     - run: pip install uv && uv sync --extra dev
   #     - run: uv run pytest -q
diff --git a/.gitignore b/.gitignore
@@ -44,5 +44,6 @@ infra/airun/generated/
 *.parquet
 
 # local junk
+.tmp/
 scratch/
 tmp/
diff --git a/Makefile b/Makefile
@@ -1,5 +1,6 @@
 .PHONY: test validate-results dashboard-index completion-report
 
+UV_RUN_DEV ?= uv run --extra dev
 RESULT_DIR ?= runs/rune/week1
 RESULT_PREFIX ?= torch-gemm
 VALIDATE_ARCH_LABELS ?= a100 h100 h200
@@ -9,9 +10,9 @@ DASHBOARD_RESULT_INDEX ?= docs/dashboard/results-index.json
 COMPLETION_REPORT ?= docs/dashboard/completion-report.md
 
 test:
-	uv run ruff format --check swordfish tests
-	uv run ruff check swordfish tests
-	uv run pytest -q
+	$(UV_RUN_DEV) ruff format --check swordfish tests
+	$(UV_RUN_DEV) ruff check swordfish tests
+	$(UV_RUN_DEV) pytest -q
 
 validate-results:
 	uv run python -m swordfish.runner validate-gemm-matrix \

diff --git a/README.md b/README.md
@@ -37,8 +37,8 @@ PyTorch/Inductor, CUTLASS/CuTe, JAX/Pallas, TileLang, and pyptx.
 ## Quick start
 
 ```bash
-uv sync
-uv run pytest
+uv sync --extra dev
+make test
 uv run python -m swordfish.runner run-gemm \
   --backend torch \
   --m 32 --n 32 --k 32 \
@@ -62,6 +62,19 @@ uv run python -m swordfish.runner bench-transformer \
   --out /tmp/swordfish-transformer-smoke.json
 ```
 
+The standalone reduction-kernel track starts with `vectorsum_v2`:
+
+```bash
+uv run python -m swordfish.runner bench-vectorsum \
+  --backend torch \
+  --size 1638400 \
+  --dtype fp32 \
+  --repeats 1 --warmup 0 --iters 1 \
+  --device cpu --allow-cpu \
+  --arch-label a100 \
+  --out /tmp/swordfish-vectorsum-smoke.json
+```
+
 To time a full training step instead of inference-only forward, use
 `--mode train-step`. This runs forward, loss, backward, and an AdamW optimizer
 step on the same tiny GPT-style reference:
@@ -93,7 +106,8 @@ uv run python -m swordfish.runner run-gemm \
 
 ### Rune setup
 
-Local CPU development only needs the base `uv sync`. Rune dispatch is opt-in:
+Local CPU development uses `uv sync --extra dev` so the optional `ruff` and
+`pytest` tools are present for `make test`. Rune dispatch is opt-in:
 `make rune-bootstrap` installs `rune-py` from the private `aks-ai-runtime`
 release tag into this repo's uv environment and then runs `rune-py bootstrap`
 to install the matching `rune` CLI into `.venv/bin`. Override
@@ -111,6 +125,7 @@ make rune-install-profiles                              # one-time symlink
 
 uv run python -m swordfish.runner list-experiments
 uv run python -m swordfish.runner explain-experiment liger-fsdp --arch a100
+uv run python -m swordfish.runner explain-experiment vectorsum-v2 --arch a100
 
 # preview the rendered Job manifest (no cluster contact)
 uv run python -m swordfish.runner submit-experiment gemm --arch h100 \
@@ -228,6 +243,10 @@ driver loader instead of pretending `torch.add` is raw PTX. Future raw-PTX
 benchmarks should plug into the same backend interface so timing, correctness,
 NCU, and JSON output do not fork per kernel.
 
+`bench-vectorsum` is the first standalone reduction target. `torch` is the fp32
+reference path; `triton` is a two-stage block reduction that writes fp32 partials
+and a scalar output while preserving the common result schema.
+
 ## License
 
 MIT
diff --git a/docs/airun/triage-log.md b/docs/airun/triage-log.md
@@ -67,3 +67,21 @@
 - Time to root cause: ~20 min
 - Fix: added Rune renderer support for `spec.runtime.securityContext.capabilities.add`, generated `swordfish-bench-a100-ncu` / `swordfish-fsdp-a100-ncu`, installed the patched local `rune`, temporarily excluded `gpu=a100` nodes from `nvidia-dcgm-exporter`, ran `swordfish-a100-ncu-rune-0502192934` with `--profile-mode ncu`, converted/fetched `profile.ncu-summary.csv`, then restored DCGM to 6/6 Ready.
 - Lesson: Profile-mode alone is not enough for A100; the easy path must select an elevated A100 NCU profile and still run inside a controlled DCGM pause window.
+
+## 2026-05-04 — A100/H200 FSDP comparison submit blocked by context and H200 capacity
+- Initial suspicion: L5
+- Actual root cause: L5 (cluster context / transient node-pool capacity) — the first Rune submit targeted the current `chokevin-aks` context, which had no `ray` namespace; after switching to `voice-agent-flex`, the first H200 comparison leg had no schedulable H200 node and hit scheduler/autoscaler max-size events.
+- Layers ruled out before finding it: L2 for the target context, because `kernel-mode-training`, `kernel-mode-large-memory`, and `team-kernel-mode-reserved-cq` existed with no initial pending workloads; L3/L4 for A100, because pinned A100 jobs admitted, scheduled to `NVIDIA-A100-SXM4-80GB`, and completed with result JSON + NSYS profiles.
+- Time to root cause: ~15 min
+- Fix: exposed `--context` and `--image` through `submit-experiment`, submitted against `voice-agent-flex`, deleted the initially blocked H200 jobs, pinned reruns to `voiceagentcr.azurecr.io/airun/swordfish-bench:bf92726-dirty` instead of cached `:dev`, and reran H200 once two Ready `NVIDIA-H200` nodes appeared.
+- Follow-up: the completed pinned comparison (`sf-fsdp-pin-{a100,h200}-*`) showed `tb-no-limit` as the best overlap lead on both A100 and H200; H200 recovered during the session, so this was a transient capacity/context blocker rather than a persistent H200 experiment blocker.
+- Lesson: For Rune sweeps, pass the kube context explicitly and pin the image tag; `:dev` plus `IfNotPresent` can reuse stale runner code even after the ACR tag has moved, and H200 must be preflighted for live schedulable nodes before using it as a comparison leg.
+
+## 2026-05-04 — vectorsum A100 capture-policy sweep pod Pending after admission
+- Initial suspicion: L3
+- Actual root cause: L3 (k8s scheduler) — `vs-v2-capture-policy-05041233` was admitted by Kueue but rendered an impossible selector: `nvidia.com/gpu.product=NVIDIA-A100-SXM4-80GB` together with `rune.ai/gpu-class=h200-nvlink-141gb`.
+- Layers ruled out before finding it: L2, because the Workload was `QuotaReserved` and `Admitted` in `team-kernel-mode-reserved-cq`.
+- Time to root cause: ~10 min
+- Fix: deleted the stuck admitted job and reran the benchmark with `--gpu-class a100-nvlink-80gb`; dry-run confirmed the selector changed to `rune.ai/gpu-class=a100-nvlink-80gb`.
+- Verification: fixed-selector reruns scheduled on `aks-gpu-33826946-vmss000001` and wrote A100 result JSON.
+- Lesson: When using a nominal A100 Rune profile, still dry-run/check the rendered `rune.ai/gpu-class`; a stale or inherited H200 GPU-class selector can make an A100 pod unschedulable even though Kueue admits it.
diff --git a/docs/research/what-can-beat-liger-fsdp-a100-20260503.md b/docs/research/what-can-beat-liger-fsdp-a100-20260503.md
@@ -0,0 +1,64 @@
+# Research: What can beat Liger for our FSDP A100 test?
+
+**Date:** 2026-05-03  
+**Asker:** Kevin / Swordfish kernel lab  
+**Decision:** ADAPT
+
+## Question
+
+For our Llama-3-8B-like bf16 FSDP1 train-step test on 8xA100, is there an existing kernel or training stack that is likely to beat Liger Kernel?
+
+## TL;DR
+
+No public source shows a clean drop-in replacement that beats Liger on the exact same Hugging Face + PyTorch FSDP1 + bf16 + 8xA100 setup. The credible "beat Liger" paths are stack changes: PyTorch FSDP2/`torch.compile`/TorchTitan for a close PyTorch-native variant, or Megatron-Core/Transformer Engine for a higher-rewrite NVIDIA stack. For Swordfish, keep Liger as the current same-test baseline and run a targeted ADAPT experiment against FSDP2/compile before considering a Megatron rewrite.
+
+## What I read
+
+| Source | Type | Date | What it said |
+|---|---|---:|---|
+| [Liger-Kernel paper](https://arxiv.org/html/2410.10989v3) | paper | 2024 | Liger reports average 20% training-throughput gain and 60% GPU-memory reduction via Triton fusion/chunking, and explicitly supports FSDP/DeepSpeed/DDP. |
+| [Liger-Kernel README](https://github.com/linkedin/Liger-Kernel) | code-repo | 2026 | Its headline benchmark is exactly close to ours: Llama-3-8B, batch 8, bf16, AdamW, gradient checkpointing, FSDP1 on 8 A100s. |
+| [PyTorch: Maximizing Training Throughput](https://pytorch.org/blog/maximizing-training-throughput/) | vendor-blog | 2024 | `torch.compile` + selective activation checkpointing raised 7B A100 MFU from 57% to 68%, with 10-23% MFU gains across model sizes. |
+| [SimpleFSDP paper](https://arxiv.org/abs/2411.00284) | paper | 2024 | Compiler-friendly FSDP can trace communication and reorder/bucket IR nodes for overlap, reporting up to 68.67% throughput improvement vs eager FSDP2 when composed with other techniques. |
+| [AWS/Meta TorchTitan Llama 3 blog](https://aws.amazon.com/blogs/machine-learning/efficient-pre-training-of-llama-3-like-model-architectures-using-torchtitan-on-amazon-sagemaker/) | vendor-blog | 2024 | TorchTitan pretrains Llama-3-8B-like models with FSDP2, `torch.compile`, and FP8, showing 38.23% throughput speedup on H100. |
+| [NVIDIA Transformer Engine docs](https://docs.nvidia.com/deeplearning/transformer-engine/user-guide/index.html) | official-docs | 2026 | TE provides optimized Transformer blocks and FP8 support on Hopper/Ada/Blackwell, plus BF16/FP16 optimizations on Ampere and later. |
+| [Megatron-LM README](https://github.com/NVIDIA/Megatron-LM) | code-repo | 2026 | Megatron-Core is a GPU-optimized training library with TP/PP/DP/CP/EP, BF16/FP8/FP4, and explicit communication-overlap optimizations. |
+| [DeepSpeed training docs](https://www.deepspeed.ai/training/) | official-docs | 2026 | DeepSpeed/ZeRO focuses on memory, communication, and scale; it can combine ZeRO data parallelism with model parallelism for speed and scale. |
+| [Unsloth multi-GPU docs](https://unsloth.ai/docs/basics/multi-gpu-training-with-unsloth) | official-docs | 2026 | Unsloth supports multi-GPU through Accelerate/DeepSpeed, but says the process is complex/manual and official multi-GPU support is still coming. |
+| [FlashAttention paper](https://arxiv.org/abs/2205.14135) | paper | 2022 | FlashAttention trains Transformers faster by IO-aware exact attention, but it is an attention kernel, not a full Llama/FSDP replacement. |
+
+(Read budget: 10 sources across paper, code-repo, official-docs, and vendor/practitioner blogs. Stopped because the answer was clear enough for a next experiment.)
+
+## Findings
+
+1. **Liger is the strongest same-shape baseline, not just a random kernel pack.** Its README names the same benchmark family as ours: Llama-3-8B, bf16, AdamW, gradient checkpointing, FSDP1, 8 A100s. The paper says the gain comes from fusion/chunking, matching our trace where memory/elementwise time fell sharply.
+
+2. **The closest credible challenger is PyTorch-native FSDP2 + `torch.compile`/TorchTitan, not another one-line kernel.** PyTorch reports 10-23% MFU gains from compile on A100 7B/13B/34B/70B runs; SimpleFSDP attacks exactly our open bottleneck by tracing collectives for compute/communication overlap. This maps directly to our exposed-NCCL problem.
+
+3. **Megatron-Core + Transformer Engine can probably beat Liger as an end-to-end training stack, but it changes the experiment.** Megatron brings tensor/pipeline/context parallelism and communication overlap; TE brings optimized transformer blocks and FP8 on newer GPUs. On A100 bf16, TE may still help, but the strongest TE story is Hopper+FP8, not our exact A100 bf16 FSDP1 row.
+
+4. **FlashAttention/xFormers are unlikely to beat Liger alone in this trace.** Our trace is already dominated by GEMM, attention, and NCCL after Liger, and attention is only one slice. FlashAttention is essential tech, but swapping attention alone is not a full replacement for Liger's RMSNorm/SwiGLU/CE fusion and will not solve exposed FSDP collectives.
+
+5. **Unsloth is not the next benchmark for this exact test.** Its public claims are strong for fine-tuning and memory efficiency, but its own docs say multi-GPU is still manual/complex. That makes it a poor immediate contender for 8xA100 full bf16 FSDP pretraining-step reproduction.
+
+## Counter-evidence
+
+The strongest counter-case is that PyTorch `torch.compile` and TorchTitan may already beat our Liger row if we port the test: PyTorch reports 7B A100 MFU rising from 57% to 68%, and SimpleFSDP claims compiler-visible collective overlap can reduce communication exposure. That is directly relevant because our Liger root trace still had fully exposed NCCL.
+
+The weakness is apples-to-apples. Those sources are not the exact Llama-3-8B/HF/FSDP1/8xA100/Liger comparison. TorchTitan's Llama-3-8B blog result is H100 with FP8 features, and SimpleFSDP compares against eager FSDP2, not Liger+FSDP1. Also, the PyTorch torchtune+Liger blog says Liger composes with `torch.compile`; if compile helps, the best result may be **Liger plus compile**, not compile instead of Liger.
+
+## Decision: ADAPT
+
+Do not replace Liger yet. Adapt the benchmark matrix to test the closest credible challengers: Liger+`torch.compile` if feasible, FSDP2/TorchTitan-style compile, and only then Megatron-Core/Transformer Engine if we are willing to change model/runtime structure.
+
+## What this means in practice
+
+- **First concrete move:** Add a Swordfish row for PyTorch-native compile/FSDP2 or TorchTitan-style Llama-3-8B 8xA100, with the same steady-state NSYS overlap analysis and tokens/sec schema.
+- **Watch-fors:** If compile/FSDP2 reduces exposed NCCL without regressing step time, it is a real challenger. If it only improves eager elementwise work, combine it with Liger instead of replacing Liger.
+- **Out of scope for this research:** H100 FP8-only wins; inference-only kernels; LoRA-only or QLoRA-only fine-tuning; convergence/quality beyond parity checks.
+
+## Open questions / what I'd read next
+
+1. Does TorchTitan/FSDP2 currently support a close Llama-3-8B bf16 8xA100 config without H100-only FP8 assumptions?
+2. Can Liger's monkey patch coexist with `torch.compile` for our exact FSDP runner without graph breaks?
+3. Does Megatron-Core have a minimal Llama-3-8B BF16 A100 recipe whose checkpoint/model semantics are close enough to compare fairly?