k8s: add nemo-training deployment (interactive, replicas:0)

[wangchen615]

Summary

Adds k8s/nemo-training-deployment.yaml: interactive NeMo Framework pod for LoRA fine-tuning and the one-shot nemo2riva conversion step.

• Image: nvcr.io/nvidia/nemo:25.04. Requires ngc-secret linked to the default SA (see CLAUDE.md § NGC Registry Access). See caveat below — this link is currently missing on NERC.
• replicas: 0 by default — scale up manually, shell in via oc rsh, run training, scale back. Matches the "scale up / use / scale down" workflow for the existing TTS deployments.
• Command: sleep infinity. No Service, no Route — access is cluster-internal via oc rsh only.
• Mounts:
  • vllm-model-cache at /models (shared with serving; also where .riva output files land for the upcoming riva-stt deployment to consume).
  • nemo-training-data at /data (training-only, from PR k8s: add nemo-training-data PVC (120Gi) #14).
  • 8Gi tmpfs at /dev/shm for PyTorch DataLoader IPC.
• Resources: 4-16 CPU, 32-64Gi RAM, 1 H100 (heavier RAM than serving pods by design).
• Standard labels (app.kubernetes.io/part-of: accent-stt, /component: training) on both the Deployment object AND the pod template (Codex-review follow-up in commit d6fff39).

What this PR does NOT cover

This PR provides the infrastructure only — a NeMo Framework pod with PVCs mounted and a GPU attached. It explicitly does not include:

• A training script (LoRA fine-tuning command, hyperparameters, config overrides).
• A data-upload procedure (oc rsync vs. huggingface-cli download, manifest format conversion between this repo's manifest.json schema and NeMo's {audio_filepath, duration, text} expectation).
• Checkpoint layout conventions (e.g., exp_manager.exp_dir=/data/experiments/<run_id>/).
• Restart-from-checkpoint instructions.
• The nemo2riva handoff procedure for converting trained checkpoints into .riva model files for the riva-stt deployment.

All of these are tracked in #18 (training runbook). That issue should be completed after at least one end-to-end training run succeeds on NERC, so the commands are validated rather than guessed.

Caveat discovered during PR authoring

oc describe sa default on the NERC project namespace currently shows only default-dockercfg-ch6wg as an image pull secret. The ngc-secret was created (46h ago, confirmed oc get secret ngc-secret) but is not linked to the default ServiceAccount. Before this Deployment can pull nvcr.io/nvidia/nemo:25.04, someone needs to run:

oc secrets link default ngc-secret --for=pull

This is the step documented in CLAUDE.md § NGC Registry Access (step 5). Flagging so the merge + apply sequence is clear.

Test plan

• uv run pre-commit run --files k8s/nemo-training-deployment.yaml passes (yamllint green).
• Link ngc-secret to default SA on NERC (oc secrets link default ngc-secret --for=pull).
• After merge, on NERC: oc apply -f k8s/nemo-training-deployment.yaml succeeds.
• oc get deployment nemo-training shows READY 0/0 (scale-to-zero by design).
• Scale up smoke test: oc scale deployment nemo-training --replicas=1 → pod reaches Running 1/1 within ~3-5 min (first pull: ~15GB image).
• oc rsh deployment/nemo-training gives a shell; nvidia-smi shows 1 H100; python -c "import nemo.collections.asr" imports cleanly.
• Pod labels confirmed: oc get pod -l app.kubernetes.io/part-of=accent-stt returns the nemo-training pod.
• Scale back down: oc scale deployment nemo-training --replicas=0.

Closes #8
Part of #5
Depends on #6 (NGC pull secret infrastructure — docs merged), #7 (PVC — merged).
Unblocks #9 (Riva deployment needs this pod to run nemo2riva once).
See also #18 (training runbook — follow-up after at least one training run succeeds).

🤖 Generated with Claude Code

[gemini-code-assist]

Important

Installation incomplete: to start using Gemini Code Assist, please ask the organization owner(s) to visit the Gemini Code Assist Admin Console and sign the Terms of Services.

[wangchen615]

Note

This review was generated by Gemini CLI.

Review for PR #15: k8s: add nemo-training deployment (interactive, replicas:0)

Technical Feedback:

• Cost Efficiency: Defaulting replicas: 0 is a smart and responsible choice for H100 GPU environments. It ensures resources are only consumed during active fine-tuning or model conversion sessions.
• DataLoader Stability: Properly mounting tmpfs at /dev/shm (8Gi) is a critical technical detail that prevents common PyTorch IPC failures during heavy data loading.
• Storage Strategy: The dual-mount strategy (shared model cache + dedicated training volume) is well-implemented.

Operational Note:

• Dependency: This deployment will be in ImagePullBackOff until PR docs: NGC registry access tutorial #13 is applied and the ngc-secret is linked to the default ServiceAccount.

Recommendation:

• Consider adding a brief note to the repository's documentation (or CLAUDE.md) with the explicit command to scale the environment up/down (oc scale deployment nemo-training --replicas=1), as the "scale-to-zero" behavior might be unexpected for new contributors.

[wangchen615]

Note

This review was generated by Codex.

• k8s/nemo-training-deployment.yaml:22-24 The new standard labels are only applied to the Deployment object, not to spec.template.metadata.labels. Deployment metadata labels do not propagate to pods, so any pod-level selection, cleanup, or observability that expects app.kubernetes.io/part-of=accent-stt or app.kubernetes.io/component=training will miss the actual training pod. If those labels are meant to be part of the new template, they should be duplicated on the pod template as well.