Several fixes for Gemma3n #39135
Several fixes for Gemma3n #39135
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| target_magnitude: torch.Tensor = torch.mean(hidden_states_0**2, dim=-1, keepdim=True) ** 0.5 | ||
| epsilon_tensor = torch.tensor(torch.finfo().min) | ||
| target_magnitude = torch.mean(hidden_states_0**2, dim=-1, keepdim=True) ** 0.5 | ||
| epsilon_tensor = torch.tensor(1e-5) |
There was a problem hiding this comment.
Seems a little bit unrelated to me, where does that come from?
There was a problem hiding this comment.
Otherwise we can get NaN on those layers because of the line current_hidden_state = current_hidden_state * (target_magnitude / torch.maximum(new_magnitude, epsilon_tensor)) - and the max ops is completely useless if we compare it to the minimum possible for any given dtype. Given the epsilon name of the variable, I assumed it is a typo and was supposed to be a small value instead for numerical stability
|
run-slow: gemma3n |
|
This comment contains run-slow, running the specified jobs: models: ['models/gemma3n'] |
|
All good, @ydshieh for the slow tests! (IntegrationTests are still skipped, I will check them soon) |
Cyrilvallez
added
the
for patch
|
Do you guys know why the training loss is exceptionally high? I don't think it's due to the gradient accumulation - it does quickly decrease, but it's very weird |
|
👀 no idea! |
|
Wait I mis-spoke - grad accumulation does in fact not work correctly. But the losses are still suspiciously high |
* remove the skips * fix the epsilon to a small value (does not make sense otherwise) * safeguard * overload test_eager_matches_sdpa * Update test_modeling_common.py * skip appropriate tests * correct no_split_layer * fix all devices issue * fix backward * fix
* remove the skips * fix the epsilon to a small value (does not make sense otherwise) * safeguard * overload test_eager_matches_sdpa * Update test_modeling_common.py * skip appropriate tests * correct no_split_layer * fix all devices issue * fix backward * fix
* remove the skips * fix the epsilon to a small value (does not make sense otherwise) * safeguard * overload test_eager_matches_sdpa * Update test_modeling_common.py * skip appropriate tests * correct no_split_layer * fix all devices issue * fix backward * fix
* remove the skips * fix the epsilon to a small value (does not make sense otherwise) * safeguard * overload test_eager_matches_sdpa * Update test_modeling_common.py * skip appropriate tests * correct no_split_layer * fix all devices issue * fix backward * fix
* remove the skips * fix the epsilon to a small value (does not make sense otherwise) * safeguard * overload test_eager_matches_sdpa * Update test_modeling_common.py * skip appropriate tests * correct no_split_layer * fix all devices issue * fix backward * fix
* remove the skips * fix the epsilon to a small value (does not make sense otherwise) * safeguard * overload test_eager_matches_sdpa * Update test_modeling_common.py * skip appropriate tests * correct no_split_layer * fix all devices issue * fix backward * fix
* remove the skips * fix the epsilon to a small value (does not make sense otherwise) * safeguard * overload test_eager_matches_sdpa * Update test_modeling_common.py * skip appropriate tests * correct no_split_layer * fix all devices issue * fix backward * fix
* remove the skips * fix the epsilon to a small value (does not make sense otherwise) * safeguard * overload test_eager_matches_sdpa * Update test_modeling_common.py * skip appropriate tests * correct no_split_layer * fix all devices issue * fix backward * fix
What does this PR do?