fix_random_seed: seed only the CPU torch RNG, not every CUDA device - #1575
Open
HenryVarro666 wants to merge 1 commit into
Open
fix_random_seed: seed only the CPU torch RNG, not every CUDA device#1575HenryVarro666 wants to merge 1 commit into
HenryVarro666 wants to merge 1 commit into
Conversation
torch.random.manual_seed() seeds the RNG of all visible CUDA devices in addition to the CPU generator. fix_random_seed() runs inside DataLoader worker processes (worker_init_fn in lhotse/dataset/dataloading.py) and in user training scripts at arbitrary points, where silently rewriting the GPU RNG state interferes with the seeding policy of the training loop. Use torch.default_generator.manual_seed() instead, which seeds only the CPU generator. CUDA seeding is the training script's responsibility (e.g. torch.cuda.manual_seed_all). Fixes lhotse-speech#1564.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1564.
Implements what was agreed in the issue:
fix_random_seed()no longer touches CUDA RNG state and seeds only the CPU torch generator (torch.default_generator.manual_seed), plusrandom, numpy and lhotse'suuid4()as before. (Following up on the thread since the PR mentioned there didn't land yet — happy to adjust if you had a different shape in mind.)Why:
torch.random.manual_seed()also seeds the RNG of every visible CUDA device in the process.fix_random_seed()runs inside DataLoader workers (worker_init_fninlhotse/dataset/dataloading.py) and at arbitrary points in user scripts, where silently rewriting GPU RNG state interferes with the training loop's own seeding policy. CUDA seeding is now explicitly the training script's responsibility, and the docstring says so.Evidence (2x H100 NVL, torch 2.8.0):
Added two tests: CPU determinism is preserved (
fix_random_seed→ identicaltorch.randndraws), and a CUDA-marked test assertingtorch.cuda.get_rng_state_all()is unchanged byfix_random_seed(passes on the 2-GPU machine, skipped without CUDA).