Skip to content

fix_random_seed: seed only the CPU torch RNG, not every CUDA device - #1575

Open
HenryVarro666 wants to merge 1 commit into
lhotse-speech:masterfrom
HenryVarro666:fix-1564-cpu-only-seed
Open

fix_random_seed: seed only the CPU torch RNG, not every CUDA device#1575
HenryVarro666 wants to merge 1 commit into
lhotse-speech:masterfrom
HenryVarro666:fix-1564-cpu-only-seed

Conversation

@HenryVarro666

Copy link
Copy Markdown

Fixes #1564.

Implements what was agreed in the issue: fix_random_seed() no longer touches CUDA RNG state and seeds only the CPU torch generator (torch.default_generator.manual_seed), plus random, numpy and lhotse's uuid4() as before. (Following up on the thread since the PR mentioned there didn't land yet — happy to adjust if you had a different shape in mind.)

Why: torch.random.manual_seed() also seeds the RNG of every visible CUDA device in the process. fix_random_seed() runs inside DataLoader workers (worker_init_fn in lhotse/dataset/dataloading.py) and at arbitrary points in user scripts, where silently rewriting GPU RNG state interferes with the training loop's own seeding policy. CUDA seeding is now explicitly the training script's responsibility, and the docstring says so.

Evidence (2x H100 NVL, torch 2.8.0):

import torch, lhotse
torch.cuda.manual_seed_all(1234)          # training script's own CUDA seeding
before = torch.cuda.get_rng_state_all()
lhotse.fix_random_seed(42)                # e.g. from a worker_init_fn
after = torch.cuda.get_rng_state_all()
# master:      CUDA RNG state changed on GPUs [0, 1]
# this branch: CUDA RNG state unchanged

Added two tests: CPU determinism is preserved (fix_random_seed → identical torch.randn draws), and a CUDA-marked test asserting torch.cuda.get_rng_state_all() is unchanged by fix_random_seed (passes on the 2-GPU machine, skipped without CUDA).

torch.random.manual_seed() seeds the RNG of all visible CUDA devices in
addition to the CPU generator. fix_random_seed() runs inside DataLoader
worker processes (worker_init_fn in lhotse/dataset/dataloading.py) and in
user training scripts at arbitrary points, where silently rewriting the GPU
RNG state interferes with the seeding policy of the training loop.

Use torch.default_generator.manual_seed() instead, which seeds only the CPU
generator. CUDA seeding is the training script's responsibility (e.g.
torch.cuda.manual_seed_all).

Fixes lhotse-speech#1564.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix_random_seed problem with devices

1 participant