Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -222,3 +222,10 @@ data/synthetic/voxtral_20260330_001/**/*.wav
# Keep small test/tuning samples as experiment evidence
# (tuning_test, tuning_test_v2, fish_clone_test, fish_speech_test,
# qwen3_clone_test, crosslingual_test are small enough to commit)

# Generated NeMo training manifests (derived from data/synthetic/*)
data/manifests/
# Resampled audio (derived from data/synthetic/*; regenerate via src/training/convert_manifest.py)
data/synthetic_16k/
# Training experiment outputs (on cluster; should not be committed)
data/experiments/
8 changes: 8 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,14 @@ httpx.post("http://localhost:8094/v1/audio/speech", json={
}, timeout=300)
```

## Training

Full workflow: [docs/training-runbook.md](docs/training-runbook.md) — covers pod boot, data upload, manifest conversion (`src/training/convert_manifest.py`), full-parameter and LoRA recipes, monitoring, checkpoint resume, and the `nemo2riva` handoff.

Quick summary: scale up `nemo-training`, run `uv run python src/training/convert_manifest.py` locally to produce NeMo-format JSONLs + resampled audio, `oc rsync` to the pod, shell in, launch training, tail logs, scale down.

Always scale back to `replicas=0` when done — H100 is ~$3/hr.

## Data & Experiment Conventions

**Run naming**: `{model_short}_{YYYYMMDD}_{NNN}` (e.g., `qwen3tts_20260330_001`)
Expand Down
343 changes: 343 additions & 0 deletions docs/training-runbook.md

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ requires-python = ">=3.10"
dependencies = [
"httpx>=0.27",
"jiwer>=3.0",
"soundfile>=0.12",
"scipy>=1.13",
"numpy>=1.26",
]

[dependency-groups]
Expand Down
22 changes: 22 additions & 0 deletions src/training/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# `src/training/`

Utilities for fine-tuning accent-adaptive ASR models on the synthetic corpus generated by `src/data_generation/`.

The end-to-end workflow (pod boot, data upload, LoRA vs full-param recipes, monitoring, checkpoint resume, `nemo2riva` handoff) lives in **[docs/training-runbook.md](../../docs/training-runbook.md)** — treat that as the authoritative guide.

## `convert_manifest.py`

Converts this repo's `data/synthetic/<run_id>/manifest.json` schema into NeMo's expected JSONL format. Handles the 24 kHz → 16 kHz resample idempotently, deterministic train/val/test split by `(voice, sentence_id)` hash, text normalization, and an optional per-accent split mode.

Typical invocation:

```bash
uv run python src/training/convert_manifest.py \
--input-manifest data/synthetic/qwen3tts_20260330_001/manifest.json \
--output-manifest data/manifests/qwen3tts_20260330_001 \
--resample-to 16000 \
--resampled-audio-dir data/synthetic_16k/qwen3tts_20260330_001 \
--split-mode three-way
```

Produces `.train.jsonl`, `.val.jsonl`, `.test.jsonl`, `.summary.json` alongside resampled audio. See `--help` for all flags (`--keep-punctuation`, `--minimal`, `--split-mode per-accent`, etc.).
Empty file added src/training/__init__.py
Empty file.
Loading