Skip to content

Commit d410a56

Browse files
authored
fix(train): align DDP no_sync with gradient accumulation (#3472)
Synchronize DDP/FSDP gradients on the final microbatch of each accumulation window. Verified with a two-rank CPU/Gloo reproduction, py_compile, diff checks, and a clean synthetic merge against current main.
1 parent 6829ba7 commit d410a56

1 file changed

Lines changed: 3 additions & 1 deletion

File tree

funasr/train_utils/trainer_ds.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -648,7 +648,9 @@ def train_epoch(
648648

649649
my_context = nullcontext
650650
if self.use_ddp or self.use_fsdp:
651-
my_context = model.no_sync if batch_idx % accum_grad != 0 else my_context
651+
my_context = (
652+
model.no_sync if (batch_idx + 1) % accum_grad != 0 else my_context
653+
)
652654
with my_context():
653655
time2 = time.perf_counter()
654656

0 commit comments

Comments
 (0)