docs: training runbook v0 + convert_manifest.py (closes #18)

[wangchen615]

Summary

Closes #18. First end-to-end training documentation for this repo, plus the manifest-conversion tool it depends on. Marked v0 UNVALIDATED — commands have not been run against NeMo 25.04 × Nemotron-Streaming on NERC yet. Bugs are expected; they become follow-up PRs.

What this PR does

• docs/training-runbook.md (~350 lines): prerequisites, data organization diagram, workflow overview, step-by-step LoRA recipe (primary), full-parameter fine-tuning as an alternative section, nemo2riva handoff to Riva serving, shutdown checklist, and a "Known unknowns" section listing every item that may need correction after the first real run (script paths, decoder type, tokenizer behavior, sample-rate assumption).
• src/training/convert_manifest.py: converts data/synthetic/<run>/manifest.json → NeMo JSONL. Key features:
  • Handles schema drift (early Qwen3 manifests lack native_lang / model / scenario).
  • Deterministic train/val/test split by MD5(voice, sentence_id) so reruns are reproducible.
  • Optional idempotent resampling (24 kHz → 16 kHz, via scipy.signal.resample_poly; skips already-resampled targets).
  • Flags: --split-mode {none|three-way|per-accent}, --keep-punctuation, --minimal, --no-write-summary.
  • Emits <stem>.summary.json with per-split / per-accent counts and total duration.
• src/training/__init__.py (empty) and src/training/README.md (short orientation pointing at the runbook).
• pyproject.toml: soundfile>=0.12, scipy>=1.13, numpy>=1.26 added as runtime deps.
• .gitignore: ignores data/manifests/, data/synthetic_16k/, data/experiments/ (all derived or runtime).
• CLAUDE.md: new Training section pointing at the runbook.

Validation status

• uv sync installs new deps cleanly.
• uv run pre-commit run passes (ruff, ruff-format, yamllint, trailing-whitespace, EOF, merge-conflict, large-file).
• Real-audio smoke test NOT yet run. The gitignored .wav files under data/synthetic/qwen3tts_20260330_001/ aren't in the local checkout. Smoke test is tracked as Smoke test convert_manifest.py with real audio on data/synthetic/* #20 — user plans to upload audio soon. Script has been structurally validated only.
• End-to-end training run on NERC has NOT been executed. The runbook explicitly admits this in its status banner.

Why land v0 now

The user is about to attempt a first NERC training run. Having a written starting point — even unvalidated — is strictly better than guessing from memory. The runbook's "Known unknowns" section and #20 (smoke test) + #18 (post-run correction) make the drift-tolerance explicit.

Related issues

• Closes Training runbook: data upload, LoRA command, checkpoint layout, restart #18 (training runbook)
• Opens Smoke test convert_manifest.py with real audio on data/synthetic/* #20 (real-audio smoke test follow-up for convert_manifest.py)
• Part of Epic Epic: Riva STT serving + NeMo training + WER benchmarking #5

🤖 Generated with Claude Code

[gemini-code-assist]

Important

Installation incomplete: to start using Gemini Code Assist, please ask the organization owner(s) to visit the Gemini Code Assist Admin Console and sign the Terms of Services.

[wangchen615]

Note

This review was generated by Codex.

• [P2] docs/training-runbook.md:302-305 The nemo2riva step currently depends on the user-local PATH. On the PR #15 pod, pip install nemo2riva will often land under /tmp/.local/bin, which is not added to PATH, so the next nemo2riva command can fail with command not found. The runbook should use python -m nemo2riva or export the user bin directory before invoking it.
• [P2] docs/training-runbook.md:162-164 The manifest rewrite hardcodes /Users/chenw615/..., so contributors using a different checkout path will leave audio_filepath pointing at the laptop path instead of /data/synthetic_16k/... inside the pod. The rewrite needs to derive the local prefix dynamically or the converter needs a configurable output root.
• [P2] docs/training-runbook.md:217-218 RUN_ID is introduced inside the interactive pod shell, but later oc exec ... /data/experiments/$RUN_ID/... commands are shown from the laptop shell where $RUN_ID is unset. That makes the copy-paste path collapse to /data/experiments//train.log, and the same scoping problem affects the later resume and nemo2riva snippets after a fresh oc rsh. The runbook should tell users to substitute a concrete run id or persist/export it before leaving the original shell.