Training runbook: data upload, LoRA command, checkpoint layout, restart

[wangchen615]

Context

PR #15 stands up a NeMo Framework pod on NERC as nemo-training (scale-to-zero, H100, mounts vllm-model-cache + nemo-training-data). What it does not cover: the actual training workflow. After merging #15 you'll have an interactive sleep infinity pod — but no documented commands for how to use it.

This issue is a runbook follow-up. It should be completed after at least one successful training run on NERC so the commands are validated, not guessed.

What to document

1. Data upload

• oc rsync ./data/synthetic/ <pod>:/data/synthetic/ for local corpora.
• huggingface-cli download <dataset-repo> --repo-type=dataset --local-dir /data/external-accented for HF-hosted datasets.
• NeMo manifest format conversion: the repo already writes manifest.json in its own schema (fields: voice, accent, native_lang, model, scenario, sentence_id, text, file, generation_time_s, file_size_kb). NeMo expects {audio_filepath, duration, text} per line. Document the conversion script (could go in src/training/convert_manifest.py).

2. LoRA training command

• Base model path: /models/nemo/nemotron-speech-streaming-en-0.6b.nemo (after huggingface-cli download nvidia/nemotron-speech-streaming-en-0.6b).
• NeMo adapter training recipe: train_asr_adapter.py (path on NeMo 25.04 image to be verified).
• Exact flags and config overrides for LoRA rank, alpha, learning rate.
• Validation split strategy.

3. Checkpoint layout

• exp_manager.exp_dir=/data/experiments/<run_id>/ — survives pod restart because /data is the PVC.
• checkpoints/last.ckpt and per-epoch epoch=*-step=*.ckpt files.
• How to reason about disk usage (a single LoRA run: ~500MB-2GB of checkpoints).

4. Restart / resume

• +init_from_ptl_ckpt=/data/experiments/<run_id>/checkpoints/last.ckpt (syntax may differ between NeMo versions; verify).
• How to inspect the checkpoint to know which epoch/step it was saved at.

5. nemo2riva handoff

• Once a LoRA run produces a merged checkpoint, nemo2riva converts .nemo → .riva for the Riva serving deployment (Add k8s/riva-stt-deployment.yaml (Riva 2.19 serving Nemotron-Streaming) #9).
• Where to drop the .riva file so riva-stt picks it up (/models/riva/<name>.riva).

Acceptance

• New file docs/training-runbook.md committed.
• CLAUDE.md "Fine-tuning Workflow" section (added in Update CLAUDE.md: scale commands, Model Status, Fine-tuning Workflow #12) points to this runbook.
• All commands validated via at least one successful end-to-end run on NERC.

Part of #5. Surfaced during PR #15 review — PR author asked "where I should upload the audio data and transcript for training? What is the training command?" and the answer was "nothing documents this yet."