Skip to content

[Trainer] tot update steps is incorrect #37777

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
2 of 4 tasks
efsotr opened this issue Apr 25, 2025 · 1 comment · May be fixed by #37923
Open
2 of 4 tasks

[Trainer] tot update steps is incorrect #37777

efsotr opened this issue Apr 25, 2025 · 1 comment · May be fixed by #37923
Labels
bug

Comments

@efsotr
Copy link
Contributor

efsotr commented Apr 25, 2025

System Info

latest main

Who can help?

@ArthurZucker

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

num_examples is the number of examples in the dataloader and is not the number of batches in the dataloader, and is not the number of batches for one process. so the remainer is incorrect.
when steps_in_epoch % args.gradient_accumulation_steps == 0 and args.gradient_accumulation_steps > 1, total_updates is one more than expected.

transformers/src/transformers/trainer.py

Lines 2497 to 2503 in 7bb619d

remainder = num_examples % args.gradient_accumulation_steps
if remainder == 0:
remainder = args.gradient_accumulation_steps
update_step = -1
total_updates = steps_in_epoch // args.gradient_accumulation_steps + 1
if args.gradient_accumulation_steps == 1:
total_updates -= 1

num_examples from:

transformers/src/transformers/trainer.py

Lines 1756 to 1768 in 7bb619d

def num_examples(self, dataloader: DataLoader) -> int:
"""
Helper to get number of samples in a [`~torch.utils.data.DataLoader`] by accessing its dataset. When
dataloader.dataset does not exist or has no length, estimates as best it can
"""
try:
dataset = dataloader.dataset
# Special case for IterableDatasetShard, we need to dig deeper
if isinstance(dataset, IterableDatasetShard):
return len(dataloader.dataset.dataset)
return len(dataloader.dataset)
except (NameError, AttributeError, TypeError): # no dataset or length, estimate by length of dataloader
return len(dataloader) * self.args.per_device_train_batch_size

Expected behavior

correct code is

 remainder = steps_in_epoch % args.gradient_accumulation_steps 
 if remainder == 0: 
     remainder = args.gradient_accumulation_steps 
 update_step = -1 
 total_updates = steps_in_epoch // args.gradient_accumulation_steps + int(remainder < args.gradient_accumulation_steps)
@efsotr efsotr added the bug label Apr 25, 2025
@efsotr efsotr mentioned this issue Apr 25, 2025
Closed
5 tasks
@Rocketknight1
Copy link
Member

cc @SunMarc

@efsotr efsotr linked a pull request May 2, 2025 that will close this issue
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix tot update in trainer efsotr/transformers
fix total updates in epoch efsotr/transformers
2 participants
@Rocketknight1 @efsotr