Skip to content
Closed
Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
a2281e3
fix(grpo,rloo): make a non-finite training step visible in log_history
behroozazarkhalili Aug 21, 2026
1764360
Merge remote-tracking branch 'origin/main' into fix/6702-nonfinite-lo…
behroozazarkhalili Aug 27, 2026
f498ed3
wip(trainers): report a non-finite loss in 20 trainers
behroozazarkhalili Aug 31, 2026
13a0666
feat: report a non-finite training step in the logged metrics
behroozazarkhalili Aug 31, 2026
5198397
Merge remote-tracking branch 'origin/main' into fix/6702-nonfinite-lo…
behroozazarkhalili Aug 31, 2026
2d67e00
fix: drop two unreachable duplicate raises in the teacher-coverage ch…
behroozazarkhalili Aug 31, 2026
6eb4686
test(nonfinite): cover Inf where it reaches the loss, fix false comments
behroozazarkhalili Sep 1, 2026
8a45fed
fix(nonfinite): pin both gather operands to float32 and correct two f…
behroozazarkhalili Sep 1, 2026
831004e
fix(nonfinite): guard PRMTrainer, and test the warning and the rate
behroozazarkhalili Sep 1, 2026
173568e
fix(trainers): make the non-finite loss guard dtype-safe and rank-safe
behroozazarkhalili Sep 2, 2026
af18bcf
Merge remote-tracking branch 'origin/main' into fix/6702-nonfinite-lo…
behroozazarkhalili Sep 2, 2026
5038703
style(tests): rewrap three docstrings to satisfy the doc-builder hook
behroozazarkhalili Sep 2, 2026
0cab401
fix(trainers): widen the loss cast and order the DPO/KTO collectives
behroozazarkhalili Sep 2, 2026
9ac794a
fix(dpo,kto): agree on a forward failure before the first metric gather
behroozazarkhalili Sep 2, 2026
822f1c4
fix(dpo,kto): agree on every forward exception and on KTO's fused loss
behroozazarkhalili Sep 2, 2026
74cd684
fix(sft): agree on every forward exception, not only ValueError
behroozazarkhalili Sep 3, 2026
9df81de
fix(sft): evaluate a dict of splits once per split, and correct three…
behroozazarkhalili Sep 3, 2026
4db11db
Merge remote-tracking branch 'origin/main' into HEAD
behroozazarkhalili Sep 3, 2026
77e5495
Merge remote-tracking branch 'origin/main' into HEAD
behroozazarkhalili Sep 3, 2026
90c9ece
Merge remote-tracking branch 'origin/main' into HEAD
behroozazarkhalili Sep 4, 2026
29c4c55
test(trainer): exercise every forward-agreement site and pin the warn…
behroozazarkhalili Sep 4, 2026
0c10d05
Merge remote-tracking branch 'origin/main' into HEAD
behroozazarkhalili Sep 4, 2026
387fac7
Merge remote-tracking branch 'origin/main' into HEAD
behroozazarkhalili Sep 4, 2026
f096119
Merge remote-tracking branch 'origin/main' into HEAD
behroozazarkhalili Sep 4, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/source/async_distillation_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,7 @@ What the objective itself measures, averaged over the trained tokens of the wind
| `teacher_entropy` | the teacher's entropy over the candidates it reported. Bounded below the true value, since only `teacher_top_k` candidates cross the wire |
| `teacher_jsd/<id>` | MOPD only: `jsd` restricted to the tokens that teacher scored. Teachers in different domains can diverge at very different rates, which the blended `jsd` conflates |
| `teacher_entropy/<id>` | MOPD only: the same breakdown of `teacher_entropy` |
| `frac_nonfinite_loss` | the fraction of loss computations in the window whose loss was not finite (NaN or Inf), one flag per rank per micro-batch. `logging_nan_inf_filter` is on by default and, when `is_torch_xla_available()` is false, hides such a step from the reported loss, so any non-zero value is worth investigating |

There is no per-teacher `entropy`: the student's entropy is a property of its own policy, not of which teacher scored the sample, so the blended metric already covers it.

Expand Down
1 change: 1 addition & 0 deletions docs/source/async_grpo_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,7 @@ The gap between them is `perf/rollout_wait_s` plus the optimizer and weight-sync
| `perf/forwarded_tok_s_fwd_bwd`, `perf/forwarded_tok_s_wall_clock` | forwarded tokens per second on each basis |
| `perf/trained_tok_s_wall_clock` | the same, counting only tokens the loss saw |
| `perf/mfu_fwd_bwd`, `perf/mfu_wall_clock` | model FLOPs utilisation on each basis |
| `frac_nonfinite_loss` | the fraction of loss computations in the window whose loss was not finite (NaN or Inf), one flag per rank per micro-batch. `logging_nan_inf_filter` is on by default and, when `is_torch_xla_available()` is false, hides such a step from the reported loss, so any non-zero value is worth investigating |

## AsyncGRPOConfig

Expand Down
1 change: 1 addition & 0 deletions docs/source/cpo_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ While training and evaluating, we record the following metrics:
* `rewards/accuracies`: mean of how often the chosen rewards are > than the corresponding rejected rewards
* `rewards/margins`: the mean difference between the chosen and corresponding rejected rewards
* `nll_loss`: the mean negative log likelihood loss of the policy model for the chosen responses
* `frac_nonfinite_loss`: The fraction of loss computations in the logging interval whose loss was not finite (NaN or Inf), counted once per rank per micro-batch, so it is a rate over loss computations rather than a count of affected optimizer steps. This trainer records it on the training path only, so it does not appear among the evaluation metrics. Any non-zero value is worth investigating: `logging_nan_inf_filter` is enabled by default and, when `is_torch_xla_available()` is false, substitutes a finite value for a non-finite loss in the reported `loss`, which leaves the failing step invisible in the loss curve.

## CPO variants

Expand Down
1 change: 1 addition & 0 deletions docs/source/distillation_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ While training and evaluating, we record the following metrics:
- `tools/call_frequency`: The average number of tool calls per completion in the generation batch. Logged only when `tools` are provided.
- `tools/failure_frequency`: The fraction of tool calls that failed (the tool was not found, raised an exception, or the call type is unsupported). It is `0.0` when no tool was called. Logged only when `tools` are provided.
- `entropy`: Average entropy of token predictions across generated completions (in nats). Not logged on the Liger fast path.
- `frac_nonfinite_loss`: The fraction of loss computations in the logging interval whose loss was not finite (NaN or Inf), counted once per rank per micro-batch, so it is a rate over loss computations rather than a count of affected optimizer steps. Any non-zero value is worth investigating: `logging_nan_inf_filter` is enabled by default and, when `is_torch_xla_available()` is false, substitutes a finite value for a non-finite loss in the reported `loss`, which leaves the failing step invisible in the loss curve.

## Customization

Expand Down
1 change: 1 addition & 0 deletions docs/source/dpo_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ While training and evaluating, we record the following metrics:
* `rewards/rejected`: The average implicit reward computed for the rejected completion, computed as \\( \beta \log \frac{\pi_{\theta}(y^{-}\!\mid x)}{\pi_{\mathrm{ref}}(y^{-}\!\mid x)} \\).
* `rewards/margins`: The average implicit reward margin between the chosen and rejected completions.
* `rewards/accuracies`: The proportion of examples where the implicit reward for the chosen completion is higher than that for the rejected completion.
* `frac_nonfinite_loss`: The fraction of loss computations in the logging interval whose loss was not finite (NaN or Inf), counted once per rank per micro-batch, so it is a rate over loss computations rather than a count of affected optimizer steps. Any non-zero value is worth investigating: `logging_nan_inf_filter` is enabled by default and, when `is_torch_xla_available()` is false, substitutes a finite value for a non-finite loss in the reported `loss`, which leaves the failing step invisible in the loss curve.

## Customization

Expand Down
1 change: 1 addition & 0 deletions docs/source/grpo_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,7 @@ While training and evaluating, we record the following metrics:
- `cispo_clip_ratio`: The ratio of token (or sequence, if `importance_sampling_level="sequence"`) importance sampling weights that were clipped at `epsilon_high` while having a positive advantage: \\(r_{i,t}(\theta) > \epsilon_\mathrm{high}\\). Logged only when `loss_type="cispo"`.
- `vespo/phi_seq_mean`: The average value of the VESPO Gamma weighting \\(\varphi(w)\\) applied to the sequence-level importance weights. Values below `1.0` mean sequences are being down-weighted. Logged only when `loss_type="vespo"`.
- `clip_ratio`: The ratio of clipped tokens reported by the fused Liger GRPO loss. Logged only when `use_liger_kernel=True`, in which case it replaces the `clip_ratio/*` metrics above.
- `frac_nonfinite_loss`: The fraction of loss computations in the logging interval whose loss was not finite (NaN or Inf), counted once per rank per micro-batch, so it is a rate over loss computations rather than a count of affected optimizer steps. Any non-zero value is worth investigating: `logging_nan_inf_filter` is enabled by default and, when `is_torch_xla_available()` is false, substitutes a finite value for a non-finite loss in the reported `loss`, which leaves the failing step invisible in the loss curve.

## Customization

Expand Down
1 change: 1 addition & 0 deletions docs/source/kto_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ While training and evaluating, we record the following metrics:
* `rewards/chosen`: The average implicit reward computed for the chosen (desirable) completion, computed as \\( \beta \log \frac{\pi_{\theta}(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} \\).
* `rewards/rejected`: The average implicit reward computed for the rejected (undesirable) completion, computed as \\( \beta \log \frac{\pi_{\theta}(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)} \\).
* `rewards/margins`: The average implicit reward margin between the chosen and rejected completions.
* `frac_nonfinite_loss`: The fraction of loss computations in the logging interval whose loss was not finite (NaN or Inf), counted once per rank per micro-batch, so it is a rate over loss computations rather than a count of affected optimizer steps. Any non-zero value is worth investigating: `logging_nan_inf_filter` is enabled by default and, when `is_torch_xla_available()` is false, substitutes a finite value for a non-finite loss in the reported `loss`, which leaves the failing step invisible in the loss curve.

## Customization

Expand Down
1 change: 1 addition & 0 deletions docs/source/nash_md_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ While training and evaluating, we record the following metrics:
* `val/ref_contain_eos_token`: The amount of times the mixture's output contains the eos token.
* `beta`: The parameter that controls the weight of the loss term representing the deviation from the reference model. Typically fixed, but can be made dynamic by passing a list to [`experimental.nash_md.NashMDConfig`].
* `mixture_coef`: Logit mixture coefficient for the model and reference model. Typically fixed, but can be made dynamic by passing a list to [`experimental.nash_md.NashMDConfig`].
* `frac_nonfinite_loss`: The fraction of loss computations in the logging interval whose loss was not finite (NaN or Inf), counted once per rank per micro-batch, so it is a rate over loss computations rather than a count of affected optimizer steps. This trainer records it on the training path only, so it does not appear among the evaluation metrics. Any non-zero value is worth investigating: `logging_nan_inf_filter` is enabled by default and, when `is_torch_xla_available()` is false, substitutes a finite value for a non-finite loss in the reported `loss`, which leaves the failing step invisible in the loss curve.

## NashMDTrainer

Expand Down
1 change: 1 addition & 0 deletions docs/source/online_dpo_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ While training and evaluating, we record the following metrics. Here is an examp
* `logps/rejected`: The mean log probabilities of the rejected completions.
* `val/contain_eos_token`: The fraction of completions which contain an EOS token.
* `beta`: The parameter that controls the weight of the loss term representing the deviation from the reference model. Typically fixed, but can be made dynamic by passing a list to [`experimental.online_dpo.OnlineDPOConfig`].
* `frac_nonfinite_loss`: The fraction of loss computations in the logging interval whose loss was not finite (NaN or Inf), counted once per rank per micro-batch, so it is a rate over loss computations rather than a count of affected optimizer steps. This trainer records it on the training path only, so it does not appear among the evaluation metrics. Any non-zero value is worth investigating: `logging_nan_inf_filter` is enabled by default and, when `is_torch_xla_available()` is false, substitutes a finite value for a non-finite loss in the reported `loss`, which leaves the failing step invisible in the loss curve.

## Benchmark experiments

Expand Down
1 change: 1 addition & 0 deletions docs/source/orpo_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@ While training and evaluating, we record the following metrics:
- `log_odds_chosen`: the mean log odds ratio of the chosen responses over the rejected responses
- `log_odds_ratio`: the mean of the `log(sigmoid(log_odds_chosen))`
- `nll_loss`: the mean negative log likelihood loss from the SFT part of the loss over chosen responses
- `frac_nonfinite_loss`: The fraction of loss computations in the logging interval whose loss was not finite (NaN or Inf), counted once per rank per micro-batch, so it is a rate over loss computations rather than a count of affected optimizer steps. This trainer records it on the training path only, so it does not appear among the evaluation metrics. Any non-zero value is worth investigating: `logging_nan_inf_filter` is enabled by default and, when `is_torch_xla_available()` is false, substitutes a finite value for a non-finite loss in the reported `loss`, which leaves the failing step invisible in the loss curve.

## ORPOTrainer

Expand Down
1 change: 1 addition & 0 deletions docs/source/reward_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ While training and evaluating, we record the following metrics:
* `margin`: The average margin (difference between chosen and rejected rewards) over the last logging interval.
* `learning_rate`: The current learning rate, which may change dynamically if a scheduler is used.
* `grad_norm`: The L2 norm of the gradients, computed before gradient clipping.
* `frac_nonfinite_loss`: The fraction of loss computations in the logging interval whose loss was not finite (NaN or Inf), counted once per rank per micro-batch, so it is a rate over loss computations rather than a count of affected optimizer steps. Any non-zero value is worth investigating: `logging_nan_inf_filter` is enabled by default and, when `is_torch_xla_available()` is false, substitutes a finite value for a non-finite loss in the reported `loss`, which leaves the failing step invisible in the loss curve.

## Customization

Expand Down
1 change: 1 addition & 0 deletions docs/source/rloo_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,7 @@ While training and evaluating, we record the following metrics:
- `clip_ratio/low_min`: The smallest per-completion clip indicator on the lower bound of the trust region: \\(r_{i,t}(\theta) < 1 - \epsilon_\mathrm{low}\\).
- `clip_ratio/high_mean`: The average ratio of sequence probabilities that were clipped on the upper bound of the trust region: \\(r_{i,t}(\theta) > 1 + \epsilon_\mathrm{high}\\).
- `clip_ratio/high_max`: The largest per-completion clip indicator on the upper bound of the trust region: \\(r_{i,t}(\theta) > 1 + \epsilon_\mathrm{high}\\).
- `frac_nonfinite_loss`: The fraction of loss computations in the logging interval whose loss was not finite (NaN or Inf), counted once per rank per micro-batch, so it is a rate over loss computations rather than a count of affected optimizer steps. Any non-zero value is worth investigating: `logging_nan_inf_filter` is enabled by default and, when `is_torch_xla_available()` is false, substitutes a finite value for a non-finite loss in the reported `loss`, which leaves the failing step invisible in the loss curve.

## Customization

Expand Down
1 change: 1 addition & 0 deletions docs/source/sft_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ While training and evaluating, we record the following metrics:
* `mean_token_accuracy`: The proportion of non-masked tokens for which the model’s top-1 prediction matches the ground truth token.
* `learning_rate`: The current learning rate, which may change dynamically if a scheduler is used.
* `grad_norm`: The L2 norm of the gradients, computed before gradient clipping.
* `frac_nonfinite_loss`: The fraction of loss computations in the logging interval whose loss was not finite (NaN or Inf), counted once per rank per micro-batch, so it is a rate over loss computations rather than a count of affected optimizer steps. Any non-zero value is worth investigating: `logging_nan_inf_filter` is enabled by default and, when `is_torch_xla_available()` is false, substitutes a finite value for a non-finite loss in the reported `loss`, which leaves the failing step invisible in the loss curve.

## Customization

Expand Down
1 change: 1 addition & 0 deletions docs/source/tpo_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ While training and evaluating, we record the following metrics:
* `rewards/rejected`: The average implicit reward computed for the rejected completion, defined as \\( \beta \log \pi_{\theta}(y^{-}\!\mid x) \\).
* `rewards/margins`: The average implicit reward margin between the chosen and rejected completions.
* `rewards/accuracies`: The proportion of examples where the implicit reward for the chosen completion is higher than that for the rejected completion.
* `frac_nonfinite_loss`: The fraction of loss computations in the logging interval whose loss was not finite (NaN or Inf), counted once per rank per micro-batch, so it is a rate over loss computations rather than a count of affected optimizer steps. Any non-zero value is worth investigating: `logging_nan_inf_filter` is enabled by default and, when `is_torch_xla_available()` is false, substitutes a finite value for a non-finite loss in the reported `loss`, which leaves the failing step invisible in the loss curve.

## TPOTrainer

Expand Down
1 change: 1 addition & 0 deletions docs/source/xpo_trainer.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ While training and evaluating, we record the following metrics:
* `val/ref_contain_eos_token`: The amount of times the reference's output contains the eos token.
* `alpha`: The weight of the XPO loss term. Typically fixed, but can be made dynamic by passing a list to [`experimental.xpo.XPOConfig`].
* `beta`: The parameter that controls the weight of the loss term representing the deviation from the reference model. Typically fixed, but can be made dynamic by passing a list to [`experimental.xpo.XPOConfig`].
* `frac_nonfinite_loss`: The fraction of loss computations in the logging interval whose loss was not finite (NaN or Inf), counted once per rank per micro-batch, so it is a rate over loss computations rather than a count of affected optimizer steps. This trainer records it on the training path only, so it does not appear among the evaluation metrics. Any non-zero value is worth investigating: `logging_nan_inf_filter` is enabled by default and, when `is_torch_xla_available()` is false, substitutes a finite value for a non-finite loss in the reported `loss`, which leaves the failing step invisible in the loss curve.

## XPOTrainer

Expand Down
Loading
Loading