Implement Multiturn support for EasyMagpie - #15764

Open

Edresson wants to merge 130 commits into

NVIDIA-NeMo:mainfrom

Edresson:nemotron_tts_multiturn_user_audio_latest

Commits on Jul 26, 2026

wip

paarthneekhara
authored and
shehzeen
committed
WIP

paarthneekhara
authored and
shehzeen
committed
speaker encoder optional

paarthneekhara
authored and
shehzeen
committed
Apply isort and black reformatting

paarthneekhara
authored and
shehzeen
committed
add option to remove text embedding and lm head

paarthneekhara
authored and
shehzeen
committed
cas encoder layers

paarthneekhara
authored and
shehzeen
committed
use IPA as text prob added during training

paarthneekhara
authored and
shehzeen
committed
Add multiturn dataloader

Edresson
authored and
shehzeen
committed
Update multiturn dataloader

Edresson
authored and
shehzeen
committed
Add multiturn config

Edresson
authored and
shehzeen
committed
Add a intermediary fix for prior

Edresson
authored and
shehzeen
committed
Fix audio tokens name

Edresson
authored and
shehzeen
committed
Add formatter to support json dataset on lhotse inference

Edresson
authored and
shehzeen
committed
Update inference to support multiturn dataloader

Edresson
authored and
shehzeen
committed
Bug fix in dataloder

Edresson
authored and
shehzeen
committed
Add parameter to remove user turns

Edresson
authored and
shehzeen
committed
Add multiturn inference script

Edresson
authored and
shehzeen
committed
Update inference script

Edresson
authored and
shehzeen
committed
Remove unused codes

Edresson
authored and
shehzeen
committed
Update inference recipe

Edresson
authored and
shehzeen
committed
Remove librosa resample

Edresson
authored and
shehzeen
committed
Add silence aug

Edresson
authored and
shehzeen
committed
Add silence tts data augmentation

Edresson
authored and
shehzeen
committed
Add parameter to remove subword text conditioning

Edresson
authored and
shehzeen
committed
Add support for extra duplex dataloaders

Edresson
authored and
shehzeen
committed
Add partial loading

Edresson
authored and
shehzeen
committed
Fix interruption handling for validation dataset

Edresson
authored and
shehzeen
committed
Add user silence mask

Edresson
authored and
shehzeen
committed
Add use_user_speaking_token

Edresson
authored and
shehzeen
committed
IPA handling in multiturn data
shehzeen
committed
Fix inference script

Edresson
authored and
shehzeen
committed
Add transition tokens on loss

Edresson
authored and
shehzeen
committed
Add extra parameter type

Edresson
authored and
shehzeen
committed
Fix augmentation

Edresson
authored and
shehzeen
committed
Add min_number_of_turns and max_gap_duration_collapse_turns

Edresson
authored and
shehzeen
committed
Add phoneme multiturn inference support and update silence augmentation

Edresson
authored and
shehzeen
committed
Fix sil augmentation on formatter

Edresson
authored and
shehzeen
committed
Fix inference

Edresson
authored and
shehzeen
committed
Add new inference script and fix data formatter

Edresson
authored and
shehzeen
committed
Fix merge issue

Edresson
authored and
shehzeen
committed
Add restore custom checkpoint to avoid full model loading on .nemo checkpoint restore

Edresson
authored and
shehzeen
committed
Add raw tts data support on TTS dataloader

Edresson
authored and
shehzeen
committed
Remove complex prefil code and add slupport to nemotron_h on prefill

Edresson
authored and
shehzeen
committed
Add user aduio conditioning

Edresson
authored and
shehzeen
committed
Add multiturn inference support with user audio conditioning

Edresson
authored and
shehzeen
committed
Add silence if user audio is not available

Edresson
authored and
shehzeen
committed
Add multiturn augmentations

Edresson
authored and
shehzeen
committed
Add update inference

Edresson
authored and
shehzeen
committed
Add use_explicit_silence_for_streaming_audio_delay

Edresson
authored and
shehzeen
committed
Add new trim augmentation

Edresson
authored and
shehzeen
committed
Add new trim aug

Edresson
authored and
shehzeen
committed
remove use_explicit_silence_for_streaming_audio_delay

Edresson
authored and
shehzeen
committed
Add new inference

Edresson
authored and
shehzeen
committed
Update inference script

Edresson
authored and
shehzeen
committed
Fix phoneme loss

Edresson
authored and
shehzeen
committed
Remove debug print

Edresson
authored and
shehzeen
committed
Fix phoneme inference

Edresson
authored and
shehzeen
committed
Add phoneme_loss_mask_padding

Edresson
authored and
shehzeen
committed
Update inference

Edresson
authored and
shehzeen
committed
Add full user prefill support on nemotron_h class

Edresson
authored and
shehzeen
committed
Add phoneme_loss_mask_agent_expanded

Edresson
authored and
shehzeen
committed
Rename phoneme_loss_mask_include_transition

Edresson
authored and
shehzeen
committed
Remove unused code

Edresson
authored and
shehzeen
committed
Add partial copy support

Edresson
authored and
shehzeen
committed
Add parameter to drop all turn in sample

Edresson
authored and
shehzeen
committed
phoneme turn dropout
shehzeen
committed
filewise metrics and aggregated metrics in the inference script
shehzeen
committed
Add multigpu inference script

Edresson
authored and
shehzeen
committed
phoneme pad for short turns
shehzeen
committed
ignore punctuation in word counting
shehzeen
committed
Add turn based metrics

Edresson
authored and
shehzeen
committed
Remove unused methods

Edresson
authored and
shehzeen
committed
Add new easymagpie compatible inference script

Edresson
authored and
shehzeen
committed
Update EasyMagpie inference script to support multiturn

Edresson
authored and
shehzeen
committed
Fix new inference volume norm

Edresson
authored and
shehzeen
committed
Expose ASR and EOU batch sizes on config

Edresson
authored and
shehzeen
committed
Get language from dataloader for multiturn eval

Edresson
authored and
shehzeen
committed
Remove old multiturn eval scripts

Edresson
authored and
shehzeen
committed
Undo unecessary changes on cutset

Edresson
authored and
shehzeen
committed
Remove unused user_audio_mask

Edresson
authored and
shehzeen
committed
Aplly Black

Edresson
authored and
shehzeen
committed
Remove unused params

Edresson
authored and
shehzeen
committed
short phoneme turn handling
shehzeen
committed
Bug fix on metrics computation

Edresson
authored and
shehzeen
committed
Add ECAPA2 SSIM

Edresson
authored and
shehzeen
committed
Revert "Add ECAPA2 SSIM"

Edresson
authored and
shehzeen
committed
Add Speaker filter

Edresson
authored and
shehzeen
committed
Add support for the text pretraining checkpoint

Edresson
authored and
shehzeen
committed
Revert "Add support for the text pretraining checkpoint"

Edresson
authored and
shehzeen
committed
Add emotion cosine similarity and emotion match rate metric

Edresson
authored and
shehzeen
committed
Update emotion_embedding_type default to score_vector

Edresson
authored and
shehzeen
committed
infererence user turn end token fix
shehzeen
committed
Update

Edresson
authored and
shehzeen
committed
Add strip_text_annotations_for_metrics parameter

Edresson
authored and
shehzeen
committed
Remove symbolic links on multiturn eval and added ground truth multiturn audio

Edresson
authored and
shehzeen
committed
Fix empty turn issue for no annotation data

Edresson
authored and
shehzeen
committed
Fix short words issue

Edresson
authored and
shehzeen
committed
Add phoneme prediction on exportable .json and .csv files

Edresson
authored and
shehzeen
committed
Fix try except errors and fix normalization

Edresson
authored and
shehzeen
committed
Clean up unused code

Edresson
authored and
shehzeen
committed
Apply Black

Edresson
authored and
shehzeen
committed
Fix imports order

Edresson
authored and
shehzeen
committed
Add unit tests

Edresson
authored and
shehzeen
committed
Fix black check

Edresson
authored and
shehzeen
committed
Replace instantiate with safe_instantiate

Edresson
authored and
shehzeen
committed
Fix unit tests

Edresson
authored and
shehzeen
committed
Add turn level volume normalization

Edresson
authored and
shehzeen
committed
remove configurable cas encoder layers since it is not needed
shehzeen
committed
Left silence pad short user audios for safety

Edresson
authored and
shehzeen
committed
cfg unconditional inference fix user channel
shehzeen
committed
preserve language texts in manifests
shehzeen
committed
docstring and setup tokenizer function reused
shehzeen
committed
Add missing docstring

Edresson
authored and
shehzeen
committed
Fix streaming_prefill method name

Edresson
authored and
shehzeen
committed
Add missing docstring

Edresson
authored and
shehzeen
committed
Fix merge issues

Edresson
authored and
shehzeen
committed
Add missing docstring

Edresson
authored and
shehzeen
committed
Drop user audio conditioning on CFG

Edresson
authored and
shehzeen
committed
text context support
shehzeen
committed
Refactor text_to_speech_dataset_lhotse_multiturn.py

Edresson
authored and
shehzeen
committed
Move helpers out of magpietts_inference.py inference example

Edresson
authored and
shehzeen
committed
Fix black formatting

Edresson
authored and
shehzeen
committed
Fix Liint checks

Edresson
authored and
shehzeen
committed

Commits on Jul 29, 2026

Merge branch 'main' into nemotron_tts_multiturn_user_audio_latest
Edresson
authored

Commits on Jul 30, 2026

undo uv.lock filechange
shehzeen
committed