Skip to content

fix: correct gradient accumulation off-by-one and lr_scheduler over-stepping #82

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
anxiangsir merged 3 commits into from
Feb 10, 2026
Merged

fix: correct gradient accumulation off-by-one and lr_scheduler over-stepping #82

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
52a1915
fix: correct gradient accumulation off-by-one and lr_scheduler over-s…
Luodian Feb 7, 2026
ab34a54
fix: align scheduler total_iters with optimizer steps under gradient …
Luodian Feb 7, 2026
18d298e
Merge branch 'main' into fix/training-loop-grad-accum-and-lr-scheduler
anxiangsir Feb 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions training/train.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,7 +350,8 @@ def _expand(name, v):
optimizer_cls = torch.optim.AdamW

opt = optimizer_cls(parameters, lr=args.lr, weight_decay=args.weight_decay)
lr_scheduler = PolynomialLRWarmup(opt, int(args.total_steps * args.warmup_ratio), args.total_steps, 2)
optimizer_total_steps = args.total_steps // args.backward_passes_per_step
lr_scheduler = PolynomialLRWarmup(opt, int(optimizer_total_steps * args.warmup_ratio), optimizer_total_steps, 2)
else:
raise ValueError(f"{args.opt} not support!")

Expand Down Expand Up @@ -652,7 +653,7 @@ def wrap_ddp(model):
list_loss.append(head_loss)
list_loss_float.append(head_loss.item())

is_accumulation_step = global_step % args.backward_passes_per_step != 0
is_accumulation_step = (global_step + 1) % args.backward_passes_per_step != 0
scaled_loss = sum(list_loss) / args.backward_passes_per_step

if is_accumulation_step:
Expand All @@ -665,8 +666,7 @@ def wrap_ddp(model):
clip_grad_norm_(pfc.parameters(), max_norm=5, norm_type=2)
opt.step()
opt.zero_grad(set_to_none=True)

lr_scheduler.step()
lr_scheduler.step()

batch_end_callback(
global_step=global_step,
Expand Down