You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Gemma 2 returns NaN when using default attn (sdpa) with padding #32390
The cause of nan is how softmax is computed over full-masked rows in the attention mask and I hope it will be fixed in future versions of PyTorch, here is a related PR
As suggested in the above issue, we can modify attn_mask to use another min value instead of torch.finfo(dtype).min, for example, torch.finfo(dtype).min / 2. To apply this, find min_dtype = torch.finfo(dtype).min in gemma modeling file and replace it with torch.finfo(dtype).min / 2.
Meanwhile, we will try to fix it on our side, thanks!
Reacted by David Chanin, XEric7, smh, yuanqian_zhao and casinca
More than this, it's expected as the sdpa path does not support logit soft-capping (For Gemma2).
We do already take into account the sdpa bug when creating the mask @qubvel see here:
Hi, I have met a problem, when I finetune Gemma2-2b using trainsformers.trainer, I find the lr is always 0, and grad_norm is nan:
so what's wrong? I using the same code to finetune llama3-8b and it works well.
This is my settings:
Hi, I have met a problem, when I finetune Gemma2-2b using trainsformers.trainer, I find the lr is always 0, and grad_norm is nan: so what's wrong? I using the same code to finetune llama3-8b and it works well. This is my settings:
Hi, I have met a problem, when I finetune Gemma2-2b using trainsformers.trainer, I find the lr is always 0, and grad_norm is nan: so what's wrong? I using the same code to finetune llama3-8b and it works well. This is my settings:
hi i have the same issue. How do you solve it? 😊
Hi, I just use eager instead of sdpa like this: model = AutoModelForCausalLM.from_pretrained(args.prune_model_path,
trust_remote_code=True, device_map=device_map, attn_implementation="eager"
)
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Activity
qubvel commentedon Aug 2, 2024
Hi @chanind, thanks for reporting the issue!
This is indeed a problem of scaled_dot_product_attention in PyTorch
scaled_dot_product_attentionproducesnans with booleanattn_maskwith zero rows. pytorch/pytorch#103963The cause of
nanis how softmax is computed over full-masked rows in the attention mask and I hope it will be fixed in future versions of PyTorch, here is a related PRAlso, a similar issue has been reported previously
Besides switching to
eager/flash_attnetion_2you could also tryfloat16dtype.attn_maskmin value.As suggested in the above issue, we can modify
attn_maskto use another min value instead oftorch.finfo(dtype).min, for example,torch.finfo(dtype).min / 2. To apply this, findmin_dtype = torch.finfo(dtype).minin gemma modeling file and replace it withtorch.finfo(dtype).min / 2.Meanwhile, we will try to fix it on our side, thanks!
Fix #32390
ArthurZucker commentedon Aug 3, 2024
More than this, it's expected as the
sdpapath does not support logit soft-capping (For Gemma2).We do already take into account the sdpa bug when creating the mask @qubvel see here:
transformers/src/transformers/models/llama/modeling_llama.py
Lines 1063 to 1072 in c1aa0ed
Which should be propagated to Gemma2. (it was not there for some reason my bad here)
ArthurZucker commentedon Aug 3, 2024
Related to #31303
qubvel commentedon Aug 3, 2024
@ArthurZucker thanks for the updated info!
yaolu-zjut commentedon Aug 9, 2024
Hi, I have met a problem, when I finetune Gemma2-2b using trainsformers.trainer, I find the lr is always 0, and grad_norm is nan:


so what's wrong? I using the same code to finetune llama3-8b and it works well.
This is my settings:
EMZEDI commentedon Aug 15, 2024
Same issue here running the code for hooking the activations of the model. Using float16 made it work.
ArthurZucker commentedon Aug 19, 2024
Hey! Make sure you are using
eagerorflash_attention_2notsdpa!Shengyun-Si commentedon Aug 22, 2024
hi i have the same issue. How do you solve it? 😊
yaolu-zjut commentedon Aug 27, 2024
Hi, I just use eager instead of sdpa like this: model = AutoModelForCausalLM.from_pretrained(args.prune_model_path,
trust_remote_code=True, device_map=device_map, attn_implementation="eager"
)
github-actions commentedon Sep 21, 2024
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
fix: switch Gemma4 configs to eager attention and right-padding
fix: switch Gemma4 configs to eager attention and right-padding
feat: add mock VLM dataset and Gemma4 pretokenize support (#1682)
feat: add mock VLM dataset and Gemma4 pretokenize support (#1682)
feat: add mock VLM dataset and Gemma4 pretokenize support (NVIDIA-NeM…