This repository was archived by the owner on Mar 15, 2024. It is now read-only.
This repository was archived by the owner on Mar 15, 2024. It is now read-only.
Gradient accumulation code #240
Description
Can you provide the gradient accumulation code? Since I only have 1 gpu for training, the max batch size that I can use is 256, and the final result can not match paper's result. So I'm thinking about doing gradient accumulation, thanks!!!