gemma.out

Initializing environment and loading dataset...
Loaded 1000 training examples and 200 evaluation examples.
Loading model and tokenizer: google/gemma-2b-it...
Model and tokenizer loaded.
Applying LoRA configuration...
trainable params: 3,686,400 || all params: 2,509,858,816 || trainable%: 0.1469
Converting pandas DataFrames to Hugging Face Datasets...
Dataset conversion complete.
Initializing GRPOTrainer...

========================================
      Starting GRPO Training with Gemma 2B      
========================================

To monitor training with TensorBoard:
1. Ensure 'tensorboard' is installed (`pip install tensorboard`)
2. In a SEPARATE terminal, navigate to the directory containing './frozen-lake-gemma-2b-it-grpo'
3. Run: tensorboard --logdir ./frozen-lake-gemma-2b-it-grpo
4. Open the URL provided (usually http://localhost:6006) in your browser.
   (If running on a remote server, you might need SSH port forwarding: ssh -N -L 6006:localhost:6006 user@server)

Training logs will also appear below...

Step 10: {'loss': -0.1387, 'grad_norm': 0.650956392288208, 'learning_rate': 2.4000000000000003e-06, 'num_tokens': 94629.0, 'completion_length': 79.43125, 'rewards/combined_reward_func': 0.46254207864403724, 'reward': 0.46254207864403724, 'reward_std': 0.21575306300073863, 'kl': 8.213828141379054e-05, 'clip_ratio': 0.0, 'epoch': 0.04}
{'loss': -0.1387, 'grad_norm': 0.650956392288208, 'learning_rate': 2.4000000000000003e-06, 'num_tokens': 94629.0, 'completion_length': 79.43125, 'rewards/combined_reward_func': 0.46254207864403724, 'reward': 0.46254207864403724, 'reward_std': 0.21575306300073863, 'kl': 8.213828141379054e-05, 'clip_ratio': 0.0, 'epoch': 0.04}
Step 20: {'loss': -0.2208, 'grad_norm': 0.6436032652854919, 'learning_rate': 4.800000000000001e-06, 'num_tokens': 188383.0, 'completion_length': 73.9625, 'rewards/combined_reward_func': 0.435223725810647, 'reward': 0.435223725810647, 'reward_std': 0.25344376972643656, 'kl': 0.0003294518165603222, 'clip_ratio': 0.0, 'epoch': 0.08}
{'loss': -0.2208, 'grad_norm': 0.6436032652854919, 'learning_rate': 4.800000000000001e-06, 'num_tokens': 188383.0, 'completion_length': 73.9625, 'rewards/combined_reward_func': 0.435223725810647, 'reward': 0.435223725810647, 'reward_std': 0.25344376972643656, 'kl': 0.0003294518165603222, 'clip_ratio': 0.0, 'epoch': 0.08}
Step 30: {'loss': -0.0625, 'grad_norm': 0.13696928322315216, 'learning_rate': 7.2e-06, 'num_tokens': 285072.0, 'completion_length': 92.30625, 'rewards/combined_reward_func': 0.4997962899506092, 'reward': 0.4997962899506092, 'reward_std': 0.12208095188252628, 'kl': 0.003077852009118942, 'clip_ratio': 0.0, 'epoch': 0.12}
{'loss': -0.0625, 'grad_norm': 0.13696928322315216, 'learning_rate': 7.2e-06, 'num_tokens': 285072.0, 'completion_length': 92.30625, 'rewards/combined_reward_func': 0.4997962899506092, 'reward': 0.4997962899506092, 'reward_std': 0.12208095188252628, 'kl': 0.003077852009118942, 'clip_ratio': 0.0, 'epoch': 0.12}
Step 40: {'loss': -0.0311, 'grad_norm': 0.35588502883911133, 'learning_rate': 9.600000000000001e-06, 'num_tokens': 381626.0, 'completion_length': 91.4625, 'rewards/combined_reward_func': 0.6165178172290325, 'reward': 0.6165178172290325, 'reward_std': 0.05085811064054724, 'kl': 0.0034969781525433065, 'clip_ratio': 0.0, 'epoch': 0.16}
{'loss': -0.0311, 'grad_norm': 0.35588502883911133, 'learning_rate': 9.600000000000001e-06, 'num_tokens': 381626.0, 'completion_length': 91.4625, 'rewards/combined_reward_func': 0.6165178172290325, 'reward': 0.6165178172290325, 'reward_std': 0.05085811064054724, 'kl': 0.0034969781525433065, 'clip_ratio': 0.0, 'epoch': 0.16}
Step 50: {'loss': -0.0324, 'grad_norm': 0.23362915217876434, 'learning_rate': 1.2e-05, 'epoch': 0.2}
{'loss': -0.0324, 'grad_norm': 0.23362915217876434, 'learning_rate': 1.2e-05, 'epoch': 0.2}
Step 50: {'eval_loss': -0.004451680928468704, 'eval_runtime': 848.3104, 'eval_samples_per_second': 0.236, 'eval_steps_per_second': 0.059, 'eval_num_tokens': 479556.0, 'eval_completion_length': 105.04875, 'eval_rewards/combined_reward_func': 0.6066244806349278, 'eval_reward': 0.6066244806349278, 'eval_reward_std': 0.06445368817076087, 'eval_kl': 0.0042117921766475774, 'eval_clip_ratio': 0.0, 'epoch': 0.2}
{'eval_loss': -0.004451680928468704, 'eval_runtime': 848.3104, 'eval_samples_per_second': 0.236, 'eval_steps_per_second': 0.059, 'eval_num_tokens': 479556.0, 'eval_completion_length': 105.04875, 'eval_rewards/combined_reward_func': 0.6066244806349278, 'eval_reward': 0.6066244806349278, 'eval_reward_std': 0.06445368817076087, 'eval_kl': 0.0042117921766475774, 'eval_clip_ratio': 0.0, 'epoch': 0.2}
Step 60: {'loss': 0.0202, 'grad_norm': 0.45295798778533936, 'learning_rate': 1.44e-05, 'num_tokens': 577550.0, 'completion_length': 100.2625, 'rewards/combined_reward_func': 0.5976596117019654, 'reward': 0.5976596117019654, 'reward_std': 0.07270264006219804, 'kl': 0.0050584652315592395, 'clip_ratio': 0.0, 'epoch': 0.24}
{'loss': 0.0202, 'grad_norm': 0.45295798778533936, 'learning_rate': 1.44e-05, 'num_tokens': 577550.0, 'completion_length': 100.2625, 'rewards/combined_reward_func': 0.5976596117019654, 'reward': 0.5976596117019654, 'reward_std': 0.07270264006219804, 'kl': 0.0050584652315592395, 'clip_ratio': 0.0, 'epoch': 0.24}
Step 70: {'loss': 0.0124, 'grad_norm': 0.40363094210624695, 'learning_rate': 1.6800000000000002e-05, 'num_tokens': 673491.0, 'completion_length': 87.63125, 'rewards/combined_reward_func': 0.6546958312392235, 'reward': 0.6546958312392235, 'reward_std': 0.08089152488391846, 'kl': 0.01041060314164497, 'clip_ratio': 0.0, 'epoch': 0.28}
{'loss': 0.0124, 'grad_norm': 0.40363094210624695, 'learning_rate': 1.6800000000000002e-05, 'num_tokens': 673491.0, 'completion_length': 87.63125, 'rewards/combined_reward_func': 0.6546958312392235, 'reward': 0.6546958312392235, 'reward_std': 0.08089152488391846, 'kl': 0.01041060314164497, 'clip_ratio': 0.0, 'epoch': 0.28}
Step 80: {'loss': -0.0111, 'grad_norm': 0.191252663731575, 'learning_rate': 1.9200000000000003e-05, 'num_tokens': 768837.0, 'completion_length': 83.9125, 'rewards/combined_reward_func': 0.6108819857239723, 'reward': 0.6108819857239723, 'reward_std': 0.09214142546989024, 'kl': 0.009850462817121297, 'clip_ratio': 0.0, 'epoch': 0.32}
{'loss': -0.0111, 'grad_norm': 0.191252663731575, 'learning_rate': 1.9200000000000003e-05, 'num_tokens': 768837.0, 'completion_length': 83.9125, 'rewards/combined_reward_func': 0.6108819857239723, 'reward': 0.6108819857239723, 'reward_std': 0.09214142546989024, 'kl': 0.009850462817121297, 'clip_ratio': 0.0, 'epoch': 0.32}
Step 90: {'loss': -0.0213, 'grad_norm': 0.2640693187713623, 'learning_rate': 2.16e-05, 'num_tokens': 863933.0, 'completion_length': 82.35, 'rewards/combined_reward_func': 0.5703030906617641, 'reward': 0.5703030906617641, 'reward_std': 0.042529996996745464, 'kl': 0.006655683979624882, 'clip_ratio': 0.0, 'epoch': 0.36}
{'loss': -0.0213, 'grad_norm': 0.2640693187713623, 'learning_rate': 2.16e-05, 'num_tokens': 863933.0, 'completion_length': 82.35, 'rewards/combined_reward_func': 0.5703030906617641, 'reward': 0.5703030906617641, 'reward_std': 0.042529996996745464, 'kl': 0.006655683979624882, 'clip_ratio': 0.0, 'epoch': 0.36}
Step 100: {'loss': -0.0091, 'grad_norm': 0.11099378019571304, 'learning_rate': 2.4e-05, 'epoch': 0.4}
{'loss': -0.0091, 'grad_norm': 0.11099378019571304, 'learning_rate': 2.4e-05, 'epoch': 0.4}
Step 100: {'eval_loss': -0.01836908422410488, 'eval_runtime': 710.122, 'eval_samples_per_second': 0.282, 'eval_steps_per_second': 0.07, 'eval_num_tokens': 959829.0, 'eval_completion_length': 88.57125, 'eval_rewards/combined_reward_func': 0.5941912749409676, 'eval_reward': 0.5941912749409676, 'eval_reward_std': 0.059010171592235566, 'eval_kl': 0.004236275291332276, 'eval_clip_ratio': 0.0, 'epoch': 0.4}
{'eval_loss': -0.01836908422410488, 'eval_runtime': 710.122, 'eval_samples_per_second': 0.282, 'eval_steps_per_second': 0.07, 'eval_num_tokens': 959829.0, 'eval_completion_length': 88.57125, 'eval_rewards/combined_reward_func': 0.5941912749409676, 'eval_reward': 0.5941912749409676, 'eval_reward_std': 0.059010171592235566, 'eval_kl': 0.004236275291332276, 'eval_clip_ratio': 0.0, 'epoch': 0.4}
Step 110: {'loss': -0.0119, 'grad_norm': 0.03187678009271622, 'learning_rate': 2.64e-05, 'num_tokens': 1056666.0, 'completion_length': 90.290625, 'rewards/combined_reward_func': 0.6025125041604042, 'reward': 0.6025125041604042, 'reward_std': 0.04760783293750137, 'kl': 0.0033380782679159893, 'clip_ratio': 0.0, 'epoch': 0.44}
{'loss': -0.0119, 'grad_norm': 0.03187678009271622, 'learning_rate': 2.64e-05, 'num_tokens': 1056666.0, 'completion_length': 90.290625, 'rewards/combined_reward_func': 0.6025125041604042, 'reward': 0.6025125041604042, 'reward_std': 0.04760783293750137, 'kl': 0.0033380782679159893, 'clip_ratio': 0.0, 'epoch': 0.44}
Step 120: {'loss': -0.0084, 'grad_norm': 0.24378244578838348, 'learning_rate': 2.88e-05, 'num_tokens': 1153703.0, 'completion_length': 94.48125, 'rewards/combined_reward_func': 0.6091791689395905, 'reward': 0.6091791689395905, 'reward_std': 0.038009228417649865, 'kl': 0.0038803605748398695, 'clip_ratio': 0.0, 'epoch': 0.48}
{'loss': -0.0084, 'grad_norm': 0.24378244578838348, 'learning_rate': 2.88e-05, 'num_tokens': 1153703.0, 'completion_length': 94.48125, 'rewards/combined_reward_func': 0.6091791689395905, 'reward': 0.6091791689395905, 'reward_std': 0.038009228417649865, 'kl': 0.0038803605748398695, 'clip_ratio': 0.0, 'epoch': 0.48}
Step 130: {'loss': -0.0147, 'grad_norm': 0.005851526279002428, 'learning_rate': 2.9998537860139564e-05, 'num_tokens': 1251456.0, 'completion_length': 98.95625, 'rewards/combined_reward_func': 0.5767667546868325, 'reward': 0.5767667546868325, 'reward_std': 0.01690271869301796, 'kl': 0.0022719104737916494, 'clip_ratio': 0.0, 'epoch': 0.52}
{'loss': -0.0147, 'grad_norm': 0.005851526279002428, 'learning_rate': 2.9998537860139564e-05, 'num_tokens': 1251456.0, 'completion_length': 98.95625, 'rewards/combined_reward_func': 0.5767667546868325, 'reward': 0.5767667546868325, 'reward_std': 0.01690271869301796, 'kl': 0.0022719104737916494, 'clip_ratio': 0.0, 'epoch': 0.52}
Step 140: {'loss': -0.0231, 'grad_norm': 0.29586926102638245, 'learning_rate': 2.9986842451482876e-05, 'num_tokens': 1348363.0, 'completion_length': 93.66875, 'rewards/combined_reward_func': 0.6380070231854915, 'reward': 0.6380070231854915, 'reward_std': 0.0405791494064033, 'kl': 0.004217687639902578, 'clip_ratio': 0.0, 'epoch': 0.56}
{'loss': -0.0231, 'grad_norm': 0.29586926102638245, 'learning_rate': 2.9986842451482876e-05, 'num_tokens': 1348363.0, 'completion_length': 93.66875, 'rewards/combined_reward_func': 0.6380070231854915, 'reward': 0.6380070231854915, 'reward_std': 0.0405791494064033, 'kl': 0.004217687639902578, 'clip_ratio': 0.0, 'epoch': 0.56}
Step 150: {'loss': -0.0393, 'grad_norm': 0.01470594760030508, 'learning_rate': 2.9963460753897364e-05, 'epoch': 0.6}
{'loss': -0.0393, 'grad_norm': 0.01470594760030508, 'learning_rate': 2.9963460753897364e-05, 'epoch': 0.6}
Step 150: {'eval_loss': -0.018579944968223572, 'eval_runtime': 828.5852, 'eval_samples_per_second': 0.241, 'eval_steps_per_second': 0.06, 'eval_num_tokens': 1447372.0, 'eval_completion_length': 102.7625, 'eval_rewards/combined_reward_func': 0.6035955406725406, 'eval_reward': 0.6035955406725406, 'eval_reward_std': 0.052916207993403075, 'eval_kl': 0.005284068080363795, 'eval_clip_ratio': 0.0, 'epoch': 0.6}
{'eval_loss': -0.018579944968223572, 'eval_runtime': 828.5852, 'eval_samples_per_second': 0.241, 'eval_steps_per_second': 0.06, 'eval_num_tokens': 1447372.0, 'eval_completion_length': 102.7625, 'eval_rewards/combined_reward_func': 0.6035955406725406, 'eval_reward': 0.6035955406725406, 'eval_reward_std': 0.052916207993403075, 'eval_kl': 0.005284068080363795, 'eval_clip_ratio': 0.0, 'epoch': 0.6}
Step 160: {'loss': -0.0253, 'grad_norm': 0.027109559625387192, 'learning_rate': 2.992841099972747e-05, 'num_tokens': 1544946.0, 'completion_length': 102.321875, 'rewards/combined_reward_func': 0.599008921906352, 'reward': 0.599008921906352, 'reward_std': 0.045520650083199145, 'kl': 0.005161404253885849, 'clip_ratio': 0.0, 'epoch': 0.64}
{'loss': -0.0253, 'grad_norm': 0.027109559625387192, 'learning_rate': 2.992841099972747e-05, 'num_tokens': 1544946.0, 'completion_length': 102.321875, 'rewards/combined_reward_func': 0.599008921906352, 'reward': 0.599008921906352, 'reward_std': 0.045520650083199145, 'kl': 0.005161404253885849, 'clip_ratio': 0.0, 'epoch': 0.64}
Step 170: {'loss': -0.0048, 'grad_norm': 0.05987275764346123, 'learning_rate': 2.988172051971717e-05, 'num_tokens': 1642360.0, 'completion_length': 96.8375, 'rewards/combined_reward_func': 0.5882098652422428, 'reward': 0.5882098652422428, 'reward_std': 0.025466464459896088, 'kl': 0.003822491253959015, 'clip_ratio': 0.0, 'epoch': 0.68}
{'loss': -0.0048, 'grad_norm': 0.05987275764346123, 'learning_rate': 2.988172051971717e-05, 'num_tokens': 1642360.0, 'completion_length': 96.8375, 'rewards/combined_reward_func': 0.5882098652422428, 'reward': 0.5882098652422428, 'reward_std': 0.025466464459896088, 'kl': 0.003822491253959015, 'clip_ratio': 0.0, 'epoch': 0.68}
Step 180: {'loss': -0.0133, 'grad_norm': 0.034943580627441406, 'learning_rate': 2.9823425721698293e-05, 'num_tokens': 1739086.0, 'completion_length': 92.5375, 'rewards/combined_reward_func': 0.6010769732296467, 'reward': 0.6010769732296467, 'reward_std': 0.021319835633039474, 'kl': 0.004797830517418333, 'clip_ratio': 0.0, 'epoch': 0.72}
{'loss': -0.0133, 'grad_norm': 0.034943580627441406, 'learning_rate': 2.9823425721698293e-05, 'num_tokens': 1739086.0, 'completion_length': 92.5375, 'rewards/combined_reward_func': 0.6010769732296467, 'reward': 0.6010769732296467, 'reward_std': 0.021319835633039474, 'kl': 0.004797830517418333, 'clip_ratio': 0.0, 'epoch': 0.72}
Step 190: {'loss': -0.0029, 'grad_norm': 0.030295759439468384, 'learning_rate': 2.975357206220079e-05, 'num_tokens': 1835544.0, 'completion_length': 90.8625, 'rewards/combined_reward_func': 0.6068940043449402, 'reward': 0.6068940043449402, 'reward_std': 0.008711171383038164, 'kl': 0.0024680935268406754, 'clip_ratio': 0.0, 'epoch': 0.76}
{'loss': -0.0029, 'grad_norm': 0.030295759439468384, 'learning_rate': 2.975357206220079e-05, 'num_tokens': 1835544.0, 'completion_length': 90.8625, 'rewards/combined_reward_func': 0.6068940043449402, 'reward': 0.6068940043449402, 'reward_std': 0.008711171383038164, 'kl': 0.0024680935268406754, 'clip_ratio': 0.0, 'epoch': 0.76}
Step 200: {'loss': -0.0135, 'grad_norm': 0.2741720676422119, 'learning_rate': 2.9672214011007087e-05, 'epoch': 0.8}
{'loss': -0.0135, 'grad_norm': 0.2741720676422119, 'learning_rate': 2.9672214011007087e-05, 'epoch': 0.8}
Step 200: {'eval_loss': 0.002049398608505726, 'eval_runtime': 704.8104, 'eval_samples_per_second': 0.284, 'eval_steps_per_second': 0.071, 'eval_num_tokens': 1931856.0, 'eval_completion_length': 91.51375, 'eval_rewards/combined_reward_func': 0.6110821095108986, 'eval_reward': 0.6110821095108986, 'eval_reward_std': 0.054492964900564406, 'eval_kl': 0.004104561717686011, 'eval_clip_ratio': 0.0, 'epoch': 0.8}
{'eval_loss': 0.002049398608505726, 'eval_runtime': 704.8104, 'eval_samples_per_second': 0.284, 'eval_steps_per_second': 0.071, 'eval_num_tokens': 1931856.0, 'eval_completion_length': 91.51375, 'eval_rewards/combined_reward_func': 0.6110821095108986, 'eval_reward': 0.6110821095108986, 'eval_reward_std': 0.054492964900564406, 'eval_kl': 0.004104561717686011, 'eval_clip_ratio': 0.0, 'epoch': 0.8}
Step 210: {'loss': -0.0033, 'grad_norm': 0.16156789660453796, 'learning_rate': 2.9579415008678196e-05, 'num_tokens': 2027780.0, 'completion_length': 88.7375, 'rewards/combined_reward_func': 0.6108572617173195, 'reward': 0.6108572617173195, 'reward_std': 0.04569460793863982, 'kl': 0.00421945892230724, 'clip_ratio': 0.0, 'epoch': 0.84}
{'loss': -0.0033, 'grad_norm': 0.16156789660453796, 'learning_rate': 2.9579415008678196e-05, 'num_tokens': 2027780.0, 'completion_length': 88.7375, 'rewards/combined_reward_func': 0.6108572617173195, 'reward': 0.6108572617173195, 'reward_std': 0.04569460793863982, 'kl': 0.00421945892230724, 'clip_ratio': 0.0, 'epoch': 0.84}
Step 220: {'loss': 0.0094, 'grad_norm': 0.220833420753479, 'learning_rate': 2.9475247417084672e-05, 'num_tokens': 2122835.0, 'completion_length': 82.09375, 'rewards/combined_reward_func': 0.5857260659337044, 'reward': 0.5857260659337044, 'reward_std': 0.04409211343154311, 'kl': 0.09675860102579463, 'clip_ratio': 0.0, 'epoch': 0.88}
{'loss': 0.0094, 'grad_norm': 0.220833420753479, 'learning_rate': 2.9475247417084672e-05, 'num_tokens': 2122835.0, 'completion_length': 82.09375, 'rewards/combined_reward_func': 0.5857260659337044, 'reward': 0.5857260659337044, 'reward_std': 0.04409211343154311, 'kl': 0.09675860102579463, 'clip_ratio': 0.0, 'epoch': 0.88}
Step 230: {'loss': -0.0226, 'grad_norm': 0.020123351365327835, 'learning_rate': 2.9359792462981007e-05, 'num_tokens': 2217415.0, 'completion_length': 79.125, 'rewards/combined_reward_func': 0.5814644798636437, 'reward': 0.5814644798636437, 'reward_std': 0.05215488052926957, 'kl': 0.008951010543387383, 'clip_ratio': 0.0, 'epoch': 0.92}
{'loss': -0.0226, 'grad_norm': 0.020123351365327835, 'learning_rate': 2.9359792462981007e-05, 'num_tokens': 2217415.0, 'completion_length': 79.125, 'rewards/combined_reward_func': 0.5814644798636437, 'reward': 0.5814644798636437, 'reward_std': 0.05215488052926957, 'kl': 0.008951010543387383, 'clip_ratio': 0.0, 'epoch': 0.92}
Step 240: {'loss': -0.0016, 'grad_norm': 0.2703430652618408, 'learning_rate': 2.923314017466745e-05, 'num_tokens': 2312854.0, 'completion_length': 84.49375, 'rewards/combined_reward_func': 0.5888056881725788, 'reward': 0.5888056881725788, 'reward_std': 0.0424462154507637, 'kl': 0.006747111867298372, 'clip_ratio': 0.0, 'epoch': 0.96}
{'loss': -0.0016, 'grad_norm': 0.2703430652618408, 'learning_rate': 2.923314017466745e-05, 'num_tokens': 2312854.0, 'completion_length': 84.49375, 'rewards/combined_reward_func': 0.5888056881725788, 'reward': 0.5888056881725788, 'reward_std': 0.0424462154507637, 'kl': 0.006747111867298372, 'clip_ratio': 0.0, 'epoch': 0.96}
Step 250: {'loss': -0.0128, 'grad_norm': 0.005357715766876936, 'learning_rate': 2.9095389311788626e-05, 'epoch': 1.0}
{'loss': -0.0128, 'grad_norm': 0.005357715766876936, 'learning_rate': 2.9095389311788626e-05, 'epoch': 1.0}
Step 250: {'eval_loss': -0.009487875737249851, 'eval_runtime': 744.6296, 'eval_samples_per_second': 0.269, 'eval_steps_per_second': 0.067, 'eval_num_tokens': 2409441.0, 'eval_completion_length': 93.48875, 'eval_rewards/combined_reward_func': 0.6058389526605606, 'eval_reward': 0.6058389526605606, 'eval_reward_std': 0.050249514980241654, 'eval_kl': 0.003561182149715023, 'eval_clip_ratio': 0.0, 'epoch': 1.0}
{'eval_loss': -0.009487875737249851, 'eval_runtime': 744.6296, 'eval_samples_per_second': 0.269, 'eval_steps_per_second': 0.067, 'eval_num_tokens': 2409441.0, 'eval_completion_length': 93.48875, 'eval_rewards/combined_reward_func': 0.6058389526605606, 'eval_reward': 0.6058389526605606, 'eval_reward_std': 0.050249514980241654, 'eval_kl': 0.003561182149715023, 'eval_clip_ratio': 0.0, 'epoch': 1.0}
Step 260: {'loss': -0.0035, 'grad_norm': 0.1884326934814453, 'learning_rate': 2.894664728832377e-05, 'num_tokens': 2506663.0, 'completion_length': 93.653125, 'rewards/combined_reward_func': 0.6234394274652004, 'reward': 0.6234394274652004, 'reward_std': 0.04645428392104804, 'kl': 0.005450455309619429, 'clip_ratio': 0.0, 'epoch': 1.04}
{'loss': -0.0035, 'grad_norm': 0.1884326934814453, 'learning_rate': 2.894664728832377e-05, 'num_tokens': 2506663.0, 'completion_length': 93.653125, 'rewards/combined_reward_func': 0.6234394274652004, 'reward': 0.6234394274652004, 'reward_std': 0.04645428392104804, 'kl': 0.005450455309619429, 'clip_ratio': 0.0, 'epoch': 1.04}
Step 270: {'loss': 0.0089, 'grad_norm': 0.003775011980906129, 'learning_rate': 2.8787030088828517e-05, 'num_tokens': 2603642.0, 'completion_length': 94.11875, 'rewards/combined_reward_func': 0.6243887588381767, 'reward': 0.6243887588381767, 'reward_std': 0.013778750644996762, 'kl': 0.0015464575444639195, 'clip_ratio': 0.0, 'epoch': 1.08}
{'loss': 0.0089, 'grad_norm': 0.003775011980906129, 'learning_rate': 2.8787030088828517e-05, 'num_tokens': 2603642.0, 'completion_length': 94.11875, 'rewards/combined_reward_func': 0.6243887588381767, 'reward': 0.6243887588381767, 'reward_std': 0.013778750644996762, 'kl': 0.0015464575444639195, 'clip_ratio': 0.0, 'epoch': 1.08}
Step 280: {'loss': 0.0006, 'grad_norm': 0.15952040255069733, 'learning_rate': 2.8616662177993633e-05, 'num_tokens': 2699253.0, 'completion_length': 85.56875, 'rewards/combined_reward_func': 0.6020553424954415, 'reward': 0.6020553424954415, 'reward_std': 0.04113257327117026, 'kl': 0.006053165374032688, 'clip_ratio': 0.0, 'epoch': 1.12}
{'loss': 0.0006, 'grad_norm': 0.15952040255069733, 'learning_rate': 2.8616662177993633e-05, 'num_tokens': 2699253.0, 'completion_length': 85.56875, 'rewards/combined_reward_func': 0.6020553424954415, 'reward': 0.6020553424954415, 'reward_std': 0.04113257327117026, 'kl': 0.006053165374032688, 'clip_ratio': 0.0, 'epoch': 1.12}
Step 290: {'loss': -0.0064, 'grad_norm': 0.15415702760219574, 'learning_rate': 2.8435676403591193e-05, 'num_tokens': 2794212.0, 'completion_length': 81.49375, 'rewards/combined_reward_func': 0.5689906813204288, 'reward': 0.5689906813204288, 'reward_std': 0.052681056628352964, 'kl': 0.006329741959052626, 'clip_ratio': 0.0, 'epoch': 1.16}
{'loss': -0.0064, 'grad_norm': 0.15415702760219574, 'learning_rate': 2.8435676403591193e-05, 'num_tokens': 2794212.0, 'completion_length': 81.49375, 'rewards/combined_reward_func': 0.5689906813204288, 'reward': 0.5689906813204288, 'reward_std': 0.052681056628352964, 'kl': 0.006329741959052626, 'clip_ratio': 0.0, 'epoch': 1.16}
Step 300: {'loss': 0.0383, 'grad_norm': 0.20304620265960693, 'learning_rate': 2.8244213892883907e-05, 'epoch': 1.2}
{'loss': 0.0383, 'grad_norm': 0.20304620265960693, 'learning_rate': 2.8244213892883907e-05, 'epoch': 1.2}
Step 300: {'eval_loss': -0.009983949363231659, 'eval_runtime': 586.1396, 'eval_samples_per_second': 0.341, 'eval_steps_per_second': 0.085, 'eval_num_tokens': 2888924.0, 'eval_completion_length': 75.42875, 'eval_rewards/combined_reward_func': 0.6247856737673283, 'eval_reward': 0.6247856737673283, 'eval_reward_std': 0.1292442858614959, 'eval_kl': 0.03137081861263141, 'eval_clip_ratio': 0.0, 'epoch': 1.2}
{'eval_loss': -0.009983949363231659, 'eval_runtime': 586.1396, 'eval_samples_per_second': 0.341, 'eval_steps_per_second': 0.085, 'eval_num_tokens': 2888924.0, 'eval_completion_length': 75.42875, 'eval_rewards/combined_reward_func': 0.6247856737673283, 'eval_reward': 0.6247856737673283, 'eval_reward_std': 0.1292442858614959, 'eval_kl': 0.03137081861263141, 'eval_clip_ratio': 0.0, 'epoch': 1.2}
Step 310: {'loss': -0.0518, 'grad_norm': 0.3358458876609802, 'learning_rate': 2.8042423942578285e-05, 'num_tokens': 2982551.0, 'completion_length': 76.559375, 'rewards/combined_reward_func': 0.5922014832496643, 'reward': 0.5922014832496643, 'reward_std': 0.10734979385742918, 'kl': 0.022236245014937595, 'clip_ratio': 0.0, 'epoch': 1.24}
{'loss': -0.0518, 'grad_norm': 0.3358458876609802, 'learning_rate': 2.8042423942578285e-05, 'num_tokens': 2982551.0, 'completion_length': 76.559375, 'rewards/combined_reward_func': 0.5922014832496643, 'reward': 0.5922014832496643, 'reward_std': 0.10734979385742918, 'kl': 0.022236245014937595, 'clip_ratio': 0.0, 'epoch': 1.24}
Step 320: {'loss': -0.0508, 'grad_norm': 0.11411988735198975, 'learning_rate': 2.78304639024076e-05, 'num_tokens': 3079901.0, 'completion_length': 96.4375, 'rewards/combined_reward_func': 0.5852642849087715, 'reward': 0.5852642849087715, 'reward_std': 0.0737482538446784, 'kl': 0.008450244062260026, 'clip_ratio': 0.0, 'epoch': 1.28}
{'loss': -0.0508, 'grad_norm': 0.11411988735198975, 'learning_rate': 2.78304639024076e-05, 'num_tokens': 3079901.0, 'completion_length': 96.4375, 'rewards/combined_reward_func': 0.5852642849087715, 'reward': 0.5852642849087715, 'reward_std': 0.0737482538446784, 'kl': 0.008450244062260026, 'clip_ratio': 0.0, 'epoch': 1.28}
Step 330: {'loss': 0.0287, 'grad_norm': 0.23946723341941833, 'learning_rate': 2.7608499052435265e-05, 'num_tokens': 3181797.0, 'completion_length': 124.85, 'rewards/combined_reward_func': 0.5827420935034752, 'reward': 0.5827420935034752, 'reward_std': 0.042134476453065874, 'kl': 0.0062441592279355975, 'clip_ratio': 0.0, 'epoch': 1.32}
{'loss': 0.0287, 'grad_norm': 0.23946723341941833, 'learning_rate': 2.7608499052435265e-05, 'num_tokens': 3181797.0, 'completion_length': 124.85, 'rewards/combined_reward_func': 0.5827420935034752, 'reward': 0.5827420935034752, 'reward_std': 0.042134476453065874, 'kl': 0.0062441592279355975, 'clip_ratio': 0.0, 'epoch': 1.32}
Step 340: {'loss': -0.0319, 'grad_norm': 0.332265168428421, 'learning_rate': 2.7376702474174428e-05, 'num_tokens': 3279665.0, 'completion_length': 99.675, 'rewards/combined_reward_func': 0.627251973003149, 'reward': 0.627251973003149, 'reward_std': 0.12756471415050327, 'kl': 0.015105101035442204, 'clip_ratio': 0.0, 'epoch': 1.3599999999999999}
{'loss': -0.0319, 'grad_norm': 0.332265168428421, 'learning_rate': 2.7376702474174428e-05, 'num_tokens': 3279665.0, 'completion_length': 99.675, 'rewards/combined_reward_func': 0.627251973003149, 'reward': 0.627251973003149, 'reward_std': 0.12756471415050327, 'kl': 0.015105101035442204, 'clip_ratio': 0.0, 'epoch': 1.36}
Step 350: {'loss': -0.0289, 'grad_norm': 0.2032882124185562, 'learning_rate': 2.7135254915624213e-05, 'epoch': 1.4}
{'loss': -0.0289, 'grad_norm': 0.2032882124185562, 'learning_rate': 2.7135254915624213e-05, 'epoch': 1.4}
Step 350: {'eval_loss': -0.005369268823415041, 'eval_runtime': 571.6348, 'eval_samples_per_second': 0.35, 'eval_steps_per_second': 0.087, 'eval_num_tokens': 3375988.0, 'eval_completion_length': 79.9175, 'eval_rewards/combined_reward_func': 0.6243264834582806, 'eval_reward': 0.6243264834582806, 'eval_reward_std': 0.087653231814038, 'eval_kl': 0.006684213783300947, 'eval_clip_ratio': 0.0, 'epoch': 1.4}
{'eval_loss': -0.005369268823415041, 'eval_runtime': 571.6348, 'eval_samples_per_second': 0.35, 'eval_steps_per_second': 0.087, 'eval_num_tokens': 3375988.0, 'eval_completion_length': 79.9175, 'eval_rewards/combined_reward_func': 0.6243264834582806, 'eval_reward': 0.6243264834582806, 'eval_reward_std': 0.087653231814038, 'eval_kl': 0.006684213783300947, 'eval_clip_ratio': 0.0, 'epoch': 1.4}
Step 360: {'loss': -0.0099, 'grad_norm': 0.36920803785324097, 'learning_rate': 2.688434465032786e-05, 'num_tokens': 3470747.0, 'completion_length': 85.13125, 'rewards/combined_reward_func': 0.6354951359331608, 'reward': 0.6354951359331608, 'reward_std': 0.1238002783851698, 'kl': 0.015179524722043425, 'clip_ratio': 0.0, 'epoch': 1.44}
{'loss': -0.0099, 'grad_norm': 0.36920803785324097, 'learning_rate': 2.688434465032786e-05, 'num_tokens': 3470747.0, 'completion_length': 85.13125, 'rewards/combined_reward_func': 0.6354951359331608, 'reward': 0.6354951359331608, 'reward_std': 0.1238002783851698, 'kl': 0.015179524722043425, 'clip_ratio': 0.0, 'epoch': 1.44}
Step 370: {'loss': -0.0128, 'grad_norm': 0.3231898248195648, 'learning_rate': 2.6624167330562697e-05, 'num_tokens': 3565076.0, 'completion_length': 77.55625, 'rewards/combined_reward_func': 0.6156871378421783, 'reward': 0.6156871378421783, 'reward_std': 0.08750752722844482, 'kl': 0.020521328372706194, 'clip_ratio': 0.0, 'epoch': 1.48}
{'loss': -0.0128, 'grad_norm': 0.3231898248195648, 'learning_rate': 2.6624167330562697e-05, 'num_tokens': 3565076.0, 'completion_length': 77.55625, 'rewards/combined_reward_func': 0.6156871378421783, 'reward': 0.6156871378421783, 'reward_std': 0.08750752722844482, 'kl': 0.020521328372706194, 'clip_ratio': 0.0, 'epoch': 1.48}
Step 380: {'loss': -0.0167, 'grad_norm': 0.20523467659950256, 'learning_rate': 2.6354925834776346e-05, 'num_tokens': 3660390.0, 'completion_length': 83.7125, 'rewards/combined_reward_func': 0.6465109929442405, 'reward': 0.6465109929442405, 'reward_std': 0.09222860033623874, 'kl': 0.009760607026692014, 'clip_ratio': 0.0, 'epoch': 1.52}
{'loss': -0.0167, 'grad_norm': 0.20523467659950256, 'learning_rate': 2.6354925834776346e-05, 'num_tokens': 3660390.0, 'completion_length': 83.7125, 'rewards/combined_reward_func': 0.6465109929442405, 'reward': 0.6465109929442405, 'reward_std': 0.09222860033623874, 'kl': 0.009760607026692014, 'clip_ratio': 0.0, 'epoch': 1.52}
Step 390: {'loss': -0.0119, 'grad_norm': 0.1945042461156845, 'learning_rate': 2.607683010938826e-05, 'num_tokens': 3756218.0, 'completion_length': 86.925, 'rewards/combined_reward_func': 0.6048106797039509, 'reward': 0.6048106797039509, 'reward_std': 0.07740937699563802, 'kl': 0.009254653101379517, 'clip_ratio': 0.0, 'epoch': 1.56}
{'loss': -0.0119, 'grad_norm': 0.1945042461156845, 'learning_rate': 2.607683010938826e-05, 'num_tokens': 3756218.0, 'completion_length': 86.925, 'rewards/combined_reward_func': 0.6048106797039509, 'reward': 0.6048106797039509, 'reward_std': 0.07740937699563802, 'kl': 0.009254653101379517, 'clip_ratio': 0.0, 'epoch': 1.56}
Step 400: {'loss': 0.0176, 'grad_norm': 0.09271735697984695, 'learning_rate': 2.5790097005079766e-05, 'epoch': 1.6}
{'loss': 0.0176, 'grad_norm': 0.09271735697984695, 'learning_rate': 2.5790097005079766e-05, 'epoch': 1.6}
Step 400: {'eval_loss': 0.0012612642021849751, 'eval_runtime': 655.8975, 'eval_samples_per_second': 0.305, 'eval_steps_per_second': 0.076, 'eval_num_tokens': 3854082.0, 'eval_completion_length': 93.35625, 'eval_rewards/combined_reward_func': 0.6323827733099461, 'eval_reward': 0.6323827733099461, 'eval_reward_std': 0.05813465462764725, 'eval_kl': 0.01690000052039977, 'eval_clip_ratio': 0.0, 'epoch': 1.6}
{'eval_loss': 0.0012612642021849751, 'eval_runtime': 655.8975, 'eval_samples_per_second': 0.305, 'eval_steps_per_second': 0.076, 'eval_num_tokens': 3854082.0, 'eval_completion_length': 93.35625, 'eval_rewards/combined_reward_func': 0.6323827733099461, 'eval_reward': 0.6323827733099461, 'eval_reward_std': 0.05813465462764725, 'eval_kl': 0.01690000052039977, 'eval_clip_ratio': 0.0, 'epoch': 1.6}
Step 410: {'loss': 0.0043, 'grad_norm': 0.33668869733810425, 'learning_rate': 2.5494950107700482e-05, 'num_tokens': 3951065.0, 'completion_length': 96.896875, 'rewards/combined_reward_func': 0.6367945276200772, 'reward': 0.6367945276200772, 'reward_std': 0.046733559132553636, 'kl': 0.013079762715642573, 'clip_ratio': 0.0, 'epoch': 1.6400000000000001}
{'loss': 0.0043, 'grad_norm': 0.33668869733810425, 'learning_rate': 2.5494950107700482e-05, 'num_tokens': 3951065.0, 'completion_length': 96.896875, 'rewards/combined_reward_func': 0.6367945276200772, 'reward': 0.6367945276200772, 'reward_std': 0.046733559132553636, 'kl': 0.013079762715642573, 'clip_ratio': 0.0, 'epoch': 1.64}
Step 420: {'loss': -0.005, 'grad_norm': 0.024093935266137123, 'learning_rate': 2.519161956392275e-05, 'num_tokens': 4047024.0, 'completion_length': 87.74375, 'rewards/combined_reward_func': 0.6152388736605644, 'reward': 0.6152388736605644, 'reward_std': 0.08327605691738427, 'kl': 0.011102473222126718, 'clip_ratio': 0.0, 'epoch': 1.6800000000000002}
{'loss': -0.005, 'grad_norm': 0.024093935266137123, 'learning_rate': 2.519161956392275e-05, 'num_tokens': 4047024.0, 'completion_length': 87.74375, 'rewards/combined_reward_func': 0.6152388736605644, 'reward': 0.6152388736605644, 'reward_std': 0.08327605691738427, 'kl': 0.011102473222126718, 'clip_ratio': 0.0, 'epoch': 1.68}
Step 430: {'loss': 0.0012, 'grad_norm': 0.13843289017677307, 'learning_rate': 2.4880341901780205e-05, 'num_tokens': 4142228.0, 'completion_length': 83.025, 'rewards/combined_reward_func': 0.6273067072033882, 'reward': 0.6273067072033882, 'reward_std': 0.06647942429408431, 'kl': 0.012642200328991748, 'clip_ratio': 0.0, 'epoch': 1.72}
{'loss': 0.0012, 'grad_norm': 0.13843289017677307, 'learning_rate': 2.4880341901780205e-05, 'num_tokens': 4142228.0, 'completion_length': 83.025, 'rewards/combined_reward_func': 0.6273067072033882, 'reward': 0.6273067072033882, 'reward_std': 0.06647942429408431, 'kl': 0.012642200328991748, 'clip_ratio': 0.0, 'epoch': 1.72}
Step 440: {'loss': -0.0041, 'grad_norm': 0.1817319393157959, 'learning_rate': 2.4561359846230346e-05, 'num_tokens': 4237884.0, 'completion_length': 85.85, 'rewards/combined_reward_func': 0.6128537781536579, 'reward': 0.6128537781536579, 'reward_std': 0.05936998071847484, 'kl': 0.014135899391840212, 'clip_ratio': 0.0, 'epoch': 1.76}
{'loss': -0.0041, 'grad_norm': 0.1817319393157959, 'learning_rate': 2.4561359846230346e-05, 'num_tokens': 4237884.0, 'completion_length': 85.85, 'rewards/combined_reward_func': 0.6128537781536579, 'reward': 0.6128537781536579, 'reward_std': 0.05936998071847484, 'kl': 0.014135899391840212, 'clip_ratio': 0.0, 'epoch': 1.76}
Step 450: {'loss': -0.0243, 'grad_norm': 0.257169634103775, 'learning_rate': 2.4234922129884873e-05, 'epoch': 1.8}
{'loss': -0.0243, 'grad_norm': 0.257169634103775, 'learning_rate': 2.4234922129884873e-05, 'epoch': 1.8}
Step 450: {'eval_loss': -0.007732365280389786, 'eval_runtime': 717.2344, 'eval_samples_per_second': 0.279, 'eval_steps_per_second': 0.07, 'eval_num_tokens': 4333650.0, 'eval_completion_length': 92.94625, 'eval_rewards/combined_reward_func': 0.6322764682769776, 'eval_reward': 0.6322764682769776, 'eval_reward_std': 0.09112055771052838, 'eval_kl': 0.021378815487842075, 'eval_clip_ratio': 0.0, 'epoch': 1.8}
{'eval_loss': -0.007732365280389786, 'eval_runtime': 717.2344, 'eval_samples_per_second': 0.279, 'eval_steps_per_second': 0.07, 'eval_num_tokens': 4333650.0, 'eval_completion_length': 92.94625, 'eval_rewards/combined_reward_func': 0.6322764682769776, 'eval_reward': 0.6322764682769776, 'eval_reward_std': 0.09112055771052838, 'eval_kl': 0.021378815487842075, 'eval_clip_ratio': 0.0, 'epoch': 1.8}
Step 460: {'loss': -0.0019, 'grad_norm': 0.2423640936613083, 'learning_rate': 2.3901283299055524e-05, 'num_tokens': 4431920.0, 'completion_length': 94.3625, 'rewards/combined_reward_func': 0.6254070907831192, 'reward': 0.6254070907831192, 'reward_std': 0.07373135446105153, 'kl': 0.017091702248580986, 'clip_ratio': 0.0, 'epoch': 1.8399999999999999}
{'loss': -0.0019, 'grad_norm': 0.2423640936613083, 'learning_rate': 2.3901283299055524e-05, 'num_tokens': 4431920.0, 'completion_length': 94.3625, 'rewards/combined_reward_func': 0.6254070907831192, 'reward': 0.6254070907831192, 'reward_std': 0.07373135446105153, 'kl': 0.017091702248580986, 'clip_ratio': 0.0, 'epoch': 1.84}
Step 470: {'loss': -0.0035, 'grad_norm': 0.35348454117774963, 'learning_rate': 2.356070351526648e-05, 'num_tokens': 4529442.0, 'completion_length': 97.5125, 'rewards/combined_reward_func': 0.6391340531408787, 'reward': 0.6391340531408787, 'reward_std': 0.09487602566368877, 'kl': 0.007549221935914829, 'clip_ratio': 0.0, 'epoch': 1.88}
{'loss': -0.0035, 'grad_norm': 0.35348454117774963, 'learning_rate': 2.356070351526648e-05, 'num_tokens': 4529442.0, 'completion_length': 97.5125, 'rewards/combined_reward_func': 0.6391340531408787, 'reward': 0.6391340531408787, 'reward_std': 0.09487602566368877, 'kl': 0.007549221935914829, 'clip_ratio': 0.0, 'epoch': 1.88}
Step 480: {'loss': -0.0002, 'grad_norm': 0.01512976922094822, 'learning_rate': 2.3213448352388256e-05, 'num_tokens': 4626993.0, 'completion_length': 97.69375, 'rewards/combined_reward_func': 0.6193112850189209, 'reward': 0.6193112850189209, 'reward_std': 0.07803040379658341, 'kl': 0.022604219573258887, 'clip_ratio': 0.0, 'epoch': 1.92}
{'loss': -0.0002, 'grad_norm': 0.01512976922094822, 'learning_rate': 2.3213448352388256e-05, 'num_tokens': 4626993.0, 'completion_length': 97.69375, 'rewards/combined_reward_func': 0.6193112850189209, 'reward': 0.6193112850189209, 'reward_std': 0.07803040379658341, 'kl': 0.022604219573258887, 'clip_ratio': 0.0, 'epoch': 1.92}
Step 490: {'loss': -0.0039, 'grad_norm': 0.2027008831501007, 'learning_rate': 2.285978858955119e-05, 'num_tokens': 4723270.0, 'completion_length': 89.73125, 'rewards/combined_reward_func': 0.6323653310537338, 'reward': 0.6323653310537338, 'reward_std': 0.08265687939710915, 'kl': 0.013746716543391813, 'clip_ratio': 0.0, 'epoch': 1.96}
{'loss': -0.0039, 'grad_norm': 0.2027008831501007, 'learning_rate': 2.285978858955119e-05, 'num_tokens': 4723270.0, 'completion_length': 89.73125, 'rewards/combined_reward_func': 0.6323653310537338, 'reward': 0.6323653310537338, 'reward_std': 0.08265687939710915, 'kl': 0.013746716543391813, 'clip_ratio': 0.0, 'epoch': 1.96}
Step 500: {'loss': -0.004, 'grad_norm': 0.1710464507341385, 'learning_rate': 2.25e-05, 'epoch': 2.0}
{'loss': -0.004, 'grad_norm': 0.1710464507341385, 'learning_rate': 2.25e-05, 'epoch': 2.0}
Step 500: {'eval_loss': -0.009279541671276093, 'eval_runtime': 617.488, 'eval_samples_per_second': 0.324, 'eval_steps_per_second': 0.081, 'eval_num_tokens': 4819682.0, 'eval_completion_length': 83.11875, 'eval_rewards/combined_reward_func': 0.6322133088111878, 'eval_reward': 0.6322133088111878, 'eval_reward_std': 0.08124686838593334, 'eval_kl': 0.01723520827887114, 'eval_clip_ratio': 0.0, 'epoch': 2.0}
{'eval_loss': -0.009279541671276093, 'eval_runtime': 617.488, 'eval_samples_per_second': 0.324, 'eval_steps_per_second': 0.081, 'eval_num_tokens': 4819682.0, 'eval_completion_length': 83.11875, 'eval_rewards/combined_reward_func': 0.6322133088111878, 'eval_reward': 0.6322133088111878, 'eval_reward_std': 0.08124686838593334, 'eval_kl': 0.01723520827887114, 'eval_clip_ratio': 0.0, 'epoch': 2.0}
Step 510: {'loss': 0.0217, 'grad_norm': 0.33062559366226196, 'learning_rate': 2.213436313605413e-05, 'num_tokens': 4915768.0, 'completion_length': 89.55625, 'rewards/combined_reward_func': 0.6125111348927021, 'reward': 0.6125111348927021, 'reward_std': 0.05041620368137956, 'kl': 0.01291897196424543, 'clip_ratio': 0.0, 'epoch': 2.04}
{'loss': 0.0217, 'grad_norm': 0.33062559366226196, 'learning_rate': 2.213436313605413e-05, 'num_tokens': 4915768.0, 'completion_length': 89.55625, 'rewards/combined_reward_func': 0.6125111348927021, 'reward': 0.6125111348927021, 'reward_std': 0.05041620368137956, 'kl': 0.01291897196424543, 'clip_ratio': 0.0, 'epoch': 2.04}
Step 520: {'loss': 0.0023, 'grad_norm': 0.10668005794286728, 'learning_rate': 2.176316311034146e-05, 'num_tokens': 5011413.0, 'completion_length': 85.78125, 'rewards/combined_reward_func': 0.5970949105918407, 'reward': 0.5970949105918407, 'reward_std': 0.07456272100098431, 'kl': 0.01803340593687608, 'clip_ratio': 0.0, 'epoch': 2.08}
{'loss': 0.0023, 'grad_norm': 0.10668005794286728, 'learning_rate': 2.176316311034146e-05, 'num_tokens': 5011413.0, 'completion_length': 85.78125, 'rewards/combined_reward_func': 0.5970949105918407, 'reward': 0.5970949105918407, 'reward_std': 0.07456272100098431, 'kl': 0.01803340593687608, 'clip_ratio': 0.0, 'epoch': 2.08}
Step 530: {'loss': -0.009, 'grad_norm': 0.162969172000885, 'learning_rate': 2.138668937347609e-05, 'num_tokens': 5105998.0, 'completion_length': 79.15625, 'rewards/combined_reward_func': 0.6248803459107876, 'reward': 0.6248803459107876, 'reward_std': 0.08855146579444409, 'kl': 0.016514232842018826, 'clip_ratio': 0.0, 'epoch': 2.12}
{'loss': -0.009, 'grad_norm': 0.162969172000885, 'learning_rate': 2.138668937347609e-05, 'num_tokens': 5105998.0, 'completion_length': 79.15625, 'rewards/combined_reward_func': 0.6248803459107876, 'reward': 0.6248803459107876, 'reward_std': 0.08855146579444409, 'kl': 0.016514232842018826, 'clip_ratio': 0.0, 'epoch': 2.12}
Step 540: {'loss': -0.0111, 'grad_norm': 0.16902509331703186, 'learning_rate': 2.100523548835343e-05, 'num_tokens': 5200997.0, 'completion_length': 81.74375, 'rewards/combined_reward_func': 0.6173061937093735, 'reward': 0.6173061937093735, 'reward_std': 0.06041070227511227, 'kl': 0.07927936916457838, 'clip_ratio': 0.0, 'epoch': 2.16}
{'loss': -0.0111, 'grad_norm': 0.16902509331703186, 'learning_rate': 2.100523548835343e-05, 'num_tokens': 5200997.0, 'completion_length': 81.74375, 'rewards/combined_reward_func': 0.6173061937093735, 'reward': 0.6173061937093735, 'reward_std': 0.06041070227511227, 'kl': 0.07927936916457838, 'clip_ratio': 0.0, 'epoch': 2.16}
Step 550: {'loss': -0.0241, 'grad_norm': 0.48602569103240967, 'learning_rate': 2.0619098901238684e-05, 'epoch': 2.2}
{'loss': -0.0241, 'grad_norm': 0.48602569103240967, 'learning_rate': 2.0619098901238684e-05, 'epoch': 2.2}
Step 550: {'eval_loss': -0.05568178743124008, 'eval_runtime': 693.6475, 'eval_samples_per_second': 0.288, 'eval_steps_per_second': 0.072, 'eval_num_tokens': 5297574.0, 'eval_completion_length': 88.06125, 'eval_rewards/combined_reward_func': 0.5987993043661117, 'eval_reward': 0.5987993043661117, 'eval_reward_std': 0.1155105333449319, 'eval_kl': 0.01779139260703232, 'eval_clip_ratio': 0.0, 'epoch': 2.2}
{'eval_loss': -0.05568178743124008, 'eval_runtime': 693.6475, 'eval_samples_per_second': 0.288, 'eval_steps_per_second': 0.072, 'eval_num_tokens': 5297574.0, 'eval_completion_length': 88.06125, 'eval_rewards/combined_reward_func': 0.5987993043661117, 'eval_reward': 0.5987993043661117, 'eval_reward_std': 0.1155105333449319, 'eval_kl': 0.01779139260703232, 'eval_clip_ratio': 0.0, 'epoch': 2.2}
Step 560: {'loss': 0.002, 'grad_norm': 0.022579725831747055, 'learning_rate': 2.022858070982723e-05, 'num_tokens': 5394631.0, 'completion_length': 93.10625, 'rewards/combined_reward_func': 0.6271804556250572, 'reward': 0.6271804556250572, 'reward_std': 0.06021227540913969, 'kl': 0.013648202924377984, 'clip_ratio': 0.0, 'epoch': 2.24}
{'loss': 0.002, 'grad_norm': 0.022579725831747055, 'learning_rate': 2.022858070982723e-05, 'num_tokens': 5394631.0, 'completion_length': 93.10625, 'rewards/combined_reward_func': 0.6271804556250572, 'reward': 0.6271804556250572, 'reward_std': 0.06021227540913969, 'kl': 0.013648202924377984, 'clip_ratio': 0.0, 'epoch': 2.24}
Step 570: {'loss': -0.0343, 'grad_norm': 0.21384185552597046, 'learning_rate': 1.983398542845767e-05, 'num_tokens': 5490702.0, 'completion_length': 88.44375, 'rewards/combined_reward_func': 0.611533485352993, 'reward': 0.611533485352993, 'reward_std': 0.09354915561852976, 'kl': 0.02395205619977787, 'clip_ratio': 0.0, 'epoch': 2.2800000000000002}
{'loss': -0.0343, 'grad_norm': 0.21384185552597046, 'learning_rate': 1.983398542845767e-05, 'num_tokens': 5490702.0, 'completion_length': 88.44375, 'rewards/combined_reward_func': 0.611533485352993, 'reward': 0.611533485352993, 'reward_std': 0.09354915561852976, 'kl': 0.02395205619977787, 'clip_ratio': 0.0, 'epoch': 2.28}
Step 580: {'loss': 0.0088, 'grad_norm': 0.18425557017326355, 'learning_rate': 1.9435620750660702e-05, 'num_tokens': 5587508.0, 'completion_length': 93.0375, 'rewards/combined_reward_func': 0.6296573787927627, 'reward': 0.6296573787927627, 'reward_std': 0.033479830529540774, 'kl': 0.01597852347404114, 'clip_ratio': 0.0, 'epoch': 2.32}
{'loss': 0.0088, 'grad_norm': 0.18425557017326355, 'learning_rate': 1.9435620750660702e-05, 'num_tokens': 5587508.0, 'completion_length': 93.0375, 'rewards/combined_reward_func': 0.6296573787927627, 'reward': 0.6296573787927627, 'reward_std': 0.033479830529540774, 'kl': 0.01597852347404114, 'clip_ratio': 0.0, 'epoch': 2.32}
Step 590: {'loss': -0.0092, 'grad_norm': 0.1988888382911682, 'learning_rate': 1.9033797309228984e-05, 'num_tokens': 5683206.0, 'completion_length': 86.1125, 'rewards/combined_reward_func': 0.6198229677975178, 'reward': 0.6198229677975178, 'reward_std': 0.10718426699750125, 'kl': 0.03864056862075813, 'clip_ratio': 0.0, 'epoch': 2.36}
{'loss': -0.0092, 'grad_norm': 0.1988888382911682, 'learning_rate': 1.9033797309228984e-05, 'num_tokens': 5683206.0, 'completion_length': 86.1125, 'rewards/combined_reward_func': 0.6198229677975178, 'reward': 0.6198229677975178, 'reward_std': 0.10718426699750125, 'kl': 0.03864056862075813, 'clip_ratio': 0.0, 'epoch': 2.36}
Step 600: {'loss': -0.0441, 'grad_norm': 0.29719188809394836, 'learning_rate': 1.8628828433995013e-05, 'epoch': 2.4}
{'loss': -0.0441, 'grad_norm': 0.29719188809394836, 'learning_rate': 1.8628828433995013e-05, 'epoch': 2.4}
Step 600: {'eval_loss': -0.028811555355787277, 'eval_runtime': 646.7607, 'eval_samples_per_second': 0.309, 'eval_steps_per_second': 0.077, 'eval_num_tokens': 5778847.0, 'eval_completion_length': 84.27125, 'eval_rewards/combined_reward_func': 0.622547600120306, 'eval_reward': 0.622547600120306, 'eval_reward_std': 0.1024309758306481, 'eval_kl': 0.020970873307960575, 'eval_clip_ratio': 0.0, 'epoch': 2.4}
{'eval_loss': -0.028811555355787277, 'eval_runtime': 646.7607, 'eval_samples_per_second': 0.309, 'eval_steps_per_second': 0.077, 'eval_num_tokens': 5778847.0, 'eval_completion_length': 84.27125, 'eval_rewards/combined_reward_func': 0.622547600120306, 'eval_reward': 0.622547600120306, 'eval_reward_std': 0.1024309758306481, 'eval_kl': 0.020970873307960575, 'eval_clip_ratio': 0.0, 'epoch': 2.4}
Step 610: {'loss': -0.0241, 'grad_norm': 0.21235251426696777, 'learning_rate': 1.822102990750595e-05, 'num_tokens': 5875045.0, 'completion_length': 87.496875, 'rewards/combined_reward_func': 0.6395075969398022, 'reward': 0.6395075969398022, 'reward_std': 0.10313090663403272, 'kl': 0.011851748483604752, 'clip_ratio': 0.0, 'epoch': 2.44}
{'loss': -0.0241, 'grad_norm': 0.21235251426696777, 'learning_rate': 1.822102990750595e-05, 'num_tokens': 5875045.0, 'completion_length': 87.496875, 'rewards/combined_reward_func': 0.6395075969398022, 'reward': 0.6395075969398022, 'reward_std': 0.10313090663403272, 'kl': 0.011851748483604752, 'clip_ratio': 0.0, 'epoch': 2.44}
Step 620: {'loss': -0.0131, 'grad_norm': 0.33313241600990295, 'learning_rate': 1.781071971878587e-05, 'num_tokens': 5971218.0, 'completion_length': 89.08125, 'rewards/combined_reward_func': 0.6345273479819298, 'reward': 0.6345273479819298, 'reward_std': 0.05288954842835665, 'kl': 0.0053624284053512385, 'clip_ratio': 0.0, 'epoch': 2.48}
{'loss': -0.0131, 'grad_norm': 0.33313241600990295, 'learning_rate': 1.781071971878587e-05, 'num_tokens': 5971218.0, 'completion_length': 89.08125, 'rewards/combined_reward_func': 0.6345273479819298, 'reward': 0.6345273479819298, 'reward_std': 0.05288954842835665, 'kl': 0.0053624284053512385, 'clip_ratio': 0.0, 'epoch': 2.48}
Step 630: {'loss': -0.0467, 'grad_norm': 0.13167650997638702, 'learning_rate': 1.7398217815377526e-05, 'num_tokens': 6067817.0, 'completion_length': 91.74375, 'rewards/combined_reward_func': 0.6051742292940616, 'reward': 0.6051742292940616, 'reward_std': 0.1087467902339995, 'kl': 0.014202181459404528, 'clip_ratio': 0.0, 'epoch': 2.52}
{'loss': -0.0467, 'grad_norm': 0.13167650997638702, 'learning_rate': 1.7398217815377526e-05, 'num_tokens': 6067817.0, 'completion_length': 91.74375, 'rewards/combined_reward_func': 0.6051742292940616, 'reward': 0.6051742292940616, 'reward_std': 0.1087467902339995, 'kl': 0.014202181459404528, 'clip_ratio': 0.0, 'epoch': 2.52}
Step 640: {'loss': -0.0085, 'grad_norm': 0.14903363585472107, 'learning_rate': 1.698384585385684e-05, 'num_tokens': 6165797.0, 'completion_length': 100.375, 'rewards/combined_reward_func': 0.5998457983136177, 'reward': 0.5998457983136177, 'reward_std': 0.05813521695090458, 'kl': 0.02616192481291364, 'clip_ratio': 0.0, 'epoch': 2.56}
{'loss': -0.0085, 'grad_norm': 0.14903363585472107, 'learning_rate': 1.698384585385684e-05, 'num_tokens': 6165797.0, 'completion_length': 100.375, 'rewards/combined_reward_func': 0.5998457983136177, 'reward': 0.5998457983136177, 'reward_std': 0.05813521695090458, 'kl': 0.02616192481291364, 'clip_ratio': 0.0, 'epoch': 2.56}
Step 650: {'loss': -0.0088, 'grad_norm': 0.30499517917633057, 'learning_rate': 1.6567926949014805e-05, 'epoch': 2.6}
{'loss': -0.0088, 'grad_norm': 0.30499517917633057, 'learning_rate': 1.6567926949014805e-05, 'epoch': 2.6}
Step 650: {'eval_loss': -0.004215485416352749, 'eval_runtime': 769.8753, 'eval_samples_per_second': 0.26, 'eval_steps_per_second': 0.065, 'eval_num_tokens': 6263035.0, 'eval_completion_length': 97.07625, 'eval_rewards/combined_reward_func': 0.636866758018732, 'eval_reward': 0.636866758018732, 'eval_reward_std': 0.08127166371792555, 'eval_kl': 0.022959124146436808, 'eval_clip_ratio': 0.0, 'epoch': 2.6}
{'eval_loss': -0.004215485416352749, 'eval_runtime': 769.8753, 'eval_samples_per_second': 0.26, 'eval_steps_per_second': 0.065, 'eval_num_tokens': 6263035.0, 'eval_completion_length': 97.07625, 'eval_rewards/combined_reward_func': 0.636866758018732, 'eval_reward': 0.636866758018732, 'eval_reward_std': 0.08127166371792555, 'eval_kl': 0.022959124146436808, 'eval_clip_ratio': 0.0, 'epoch': 2.6}
Step 660: {'loss': -0.0199, 'grad_norm': 0.21539729833602905, 'learning_rate': 1.615078542190228e-05, 'num_tokens': 6360094.0, 'completion_length': 95.178125, 'rewards/combined_reward_func': 0.6225758284330368, 'reward': 0.6225758284330368, 'reward_std': 0.08815381931490265, 'kl': 0.022615489004965638, 'clip_ratio': 0.0, 'epoch': 2.64}
{'loss': -0.0199, 'grad_norm': 0.21539729833602905, 'learning_rate': 1.615078542190228e-05, 'num_tokens': 6360094.0, 'completion_length': 95.178125, 'rewards/combined_reward_func': 0.6225758284330368, 'reward': 0.6225758284330368, 'reward_std': 0.08815381931490265, 'kl': 0.022615489004965638, 'clip_ratio': 0.0, 'epoch': 2.64}
Step 670: {'loss': -0.0159, 'grad_norm': 0.24039635062217712, 'learning_rate': 1.57327465469342e-05, 'num_tokens': 6458455.0, 'completion_length': 102.75625, 'rewards/combined_reward_func': 0.6354777410626411, 'reward': 0.6354777410626411, 'reward_std': 0.08384958594106137, 'kl': 0.011474811659718398, 'clip_ratio': 0.0, 'epoch': 2.68}
{'loss': -0.0159, 'grad_norm': 0.24039635062217712, 'learning_rate': 1.57327465469342e-05, 'num_tokens': 6458455.0, 'completion_length': 102.75625, 'rewards/combined_reward_func': 0.6354777410626411, 'reward': 0.6354777410626411, 'reward_std': 0.08384958594106137, 'kl': 0.011474811659718398, 'clip_ratio': 0.0, 'epoch': 2.68}
Step 680: {'loss': -0.0075, 'grad_norm': 0.139334037899971, 'learning_rate': 1.5314136298250355e-05, 'num_tokens': 6556172.0, 'completion_length': 98.73125, 'rewards/combined_reward_func': 0.6431552097201347, 'reward': 0.6431552097201347, 'reward_std': 0.0961258452385664, 'kl': 0.012917198998184175, 'clip_ratio': 0.0, 'epoch': 2.7199999999999998}
{'loss': -0.0075, 'grad_norm': 0.139334037899971, 'learning_rate': 1.5314136298250355e-05, 'num_tokens': 6556172.0, 'completion_length': 98.73125, 'rewards/combined_reward_func': 0.6431552097201347, 'reward': 0.6431552097201347, 'reward_std': 0.0961258452385664, 'kl': 0.012917198998184175, 'clip_ratio': 0.0, 'epoch': 2.72}
Step 690: {'loss': -0.0166, 'grad_norm': 0.12488842755556107, 'learning_rate': 1.4895281095530577e-05, 'num_tokens': 6652791.0, 'completion_length': 91.86875, 'rewards/combined_reward_func': 0.6238492414355278, 'reward': 0.6238492414355278, 'reward_std': 0.06606200863607228, 'kl': 0.02096110999264056, 'clip_ratio': 0.0, 'epoch': 2.76}
{'loss': -0.0166, 'grad_norm': 0.12488842755556107, 'learning_rate': 1.4895281095530577e-05, 'num_tokens': 6652791.0, 'completion_length': 91.86875, 'rewards/combined_reward_func': 0.6238492414355278, 'reward': 0.6238492414355278, 'reward_std': 0.06606200863607228, 'kl': 0.02096110999264056, 'clip_ratio': 0.0, 'epoch': 2.76}
Step 700: {'loss': -0.0243, 'grad_norm': 0.021187659353017807, 'learning_rate': 1.447650754946249e-05, 'epoch': 2.8}
{'loss': -0.0243, 'grad_norm': 0.021187659353017807, 'learning_rate': 1.447650754946249e-05, 'epoch': 2.8}
Step 700: {'eval_loss': -0.010964499786496162, 'eval_runtime': 652.7456, 'eval_samples_per_second': 0.306, 'eval_steps_per_second': 0.077, 'eval_num_tokens': 6747744.0, 'eval_completion_length': 88.15125, 'eval_rewards/combined_reward_func': 0.6354639099538326, 'eval_reward': 0.6354639099538326, 'eval_reward_std': 0.06081733707804233, 'eval_kl': 0.01650115119628026, 'eval_clip_ratio': 0.0, 'epoch': 2.8}
{'eval_loss': -0.010964499786496162, 'eval_runtime': 652.7456, 'eval_samples_per_second': 0.306, 'eval_steps_per_second': 0.077, 'eval_num_tokens': 6747744.0, 'eval_completion_length': 88.15125, 'eval_rewards/combined_reward_func': 0.6354639099538326, 'eval_reward': 0.6354639099538326, 'eval_reward_std': 0.06081733707804233, 'eval_kl': 0.01650115119628026, 'eval_clip_ratio': 0.0, 'epoch': 2.8}
Step 710: {'loss': -0.0169, 'grad_norm': 0.23429344594478607, 'learning_rate': 1.40581422070603e-05, 'num_tokens': 6844449.0, 'completion_length': 86.93125, 'rewards/combined_reward_func': 0.6035723935812711, 'reward': 0.6035723935812711, 'reward_std': 0.08093548183096573, 'kl': 0.018663832878883114, 'clip_ratio': 0.0, 'epoch': 2.84}
{'loss': -0.0169, 'grad_norm': 0.23429344594478607, 'learning_rate': 1.40581422070603e-05, 'num_tokens': 6844449.0, 'completion_length': 86.93125, 'rewards/combined_reward_func': 0.6035723935812711, 'reward': 0.6035723935812711, 'reward_std': 0.08093548183096573, 'kl': 0.018663832878883114, 'clip_ratio': 0.0, 'epoch': 2.84}
Step 720: {'loss': -0.0099, 'grad_norm': 0.3094883859157562, 'learning_rate': 1.36405112970333e-05, 'num_tokens': 6940739.0, 'completion_length': 89.8125, 'rewards/combined_reward_func': 0.6271252050995827, 'reward': 0.6271252050995827, 'reward_std': 0.05285457104910165, 'kl': 0.009176869494694984, 'clip_ratio': 0.0, 'epoch': 2.88}
{'loss': -0.0099, 'grad_norm': 0.3094883859157562, 'learning_rate': 1.36405112970333e-05, 'num_tokens': 6940739.0, 'completion_length': 89.8125, 'rewards/combined_reward_func': 0.6271252050995827, 'reward': 0.6271252050995827, 'reward_std': 0.05285457104910165, 'kl': 0.009176869494694984, 'clip_ratio': 0.0, 'epoch': 2.88}
Step 730: {'loss': -0.0191, 'grad_norm': 0.14991015195846558, 'learning_rate': 1.3223940475402485e-05, 'num_tokens': 7037690.0, 'completion_length': 93.94375, 'rewards/combined_reward_func': 0.6450643911957741, 'reward': 0.6450643911957741, 'reward_std': 0.05479726195335388, 'kl': 0.010261288911715382, 'clip_ratio': 0.0, 'epoch': 2.92}
{'loss': -0.0191, 'grad_norm': 0.14991015195846558, 'learning_rate': 1.3223940475402485e-05, 'num_tokens': 7037690.0, 'completion_length': 93.94375, 'rewards/combined_reward_func': 0.6450643911957741, 'reward': 0.6450643911957741, 'reward_std': 0.05479726195335388, 'kl': 0.010261288911715382, 'clip_ratio': 0.0, 'epoch': 2.92}
Step 740: {'loss': 0.0014, 'grad_norm': 0.19004690647125244, 'learning_rate': 1.2808754571563827e-05, 'num_tokens': 7135695.0, 'completion_length': 100.53125, 'rewards/combined_reward_func': 0.624671933054924, 'reward': 0.624671933054924, 'reward_std': 0.05062911461573094, 'kl': 0.011196531771565788, 'clip_ratio': 0.0, 'epoch': 2.96}
{'loss': 0.0014, 'grad_norm': 0.19004690647125244, 'learning_rate': 1.2808754571563827e-05, 'num_tokens': 7135695.0, 'completion_length': 100.53125, 'rewards/combined_reward_func': 0.624671933054924, 'reward': 0.624671933054924, 'reward_std': 0.05062911461573094, 'kl': 0.011196531771565788, 'clip_ratio': 0.0, 'epoch': 2.96}
Step 750: {'loss': 0.0032, 'grad_norm': 0.14902867376804352, 'learning_rate': 1.2395277334996045e-05, 'epoch': 3.0}
{'loss': 0.0032, 'grad_norm': 0.14902867376804352, 'learning_rate': 1.2395277334996045e-05, 'epoch': 3.0}
Step 750: {'eval_loss': -0.019453704357147217, 'eval_runtime': 735.1023, 'eval_samples_per_second': 0.272, 'eval_steps_per_second': 0.068, 'eval_num_tokens': 7233052.0, 'eval_completion_length': 94.695, 'eval_rewards/combined_reward_func': 0.6396290212869644, 'eval_reward': 0.6396290212869644, 'eval_reward_std': 0.08270836055278778, 'eval_kl': 0.015311110607726733, 'eval_clip_ratio': 0.0, 'epoch': 3.0}
{'eval_loss': -0.019453704357147217, 'eval_runtime': 735.1023, 'eval_samples_per_second': 0.272, 'eval_steps_per_second': 0.068, 'eval_num_tokens': 7233052.0, 'eval_completion_length': 94.695, 'eval_rewards/combined_reward_func': 0.6396290212869644, 'eval_reward': 0.6396290212869644, 'eval_reward_std': 0.08270836055278778, 'eval_kl': 0.015311110607726733, 'eval_clip_ratio': 0.0, 'epoch': 3.0}
Step 760: {'loss': 0.0019, 'grad_norm': 0.1353251039981842, 'learning_rate': 1.1983831182810534e-05, 'num_tokens': 7331324.0, 'completion_length': 99.340625, 'rewards/combined_reward_func': 0.6352740336209536, 'reward': 0.6352740336209536, 'reward_std': 0.038073176372563466, 'kl': 0.020635014398067142, 'clip_ratio': 0.0, 'epoch': 3.04}
{'loss': 0.0019, 'grad_norm': 0.1353251039981842, 'learning_rate': 1.1983831182810534e-05, 'num_tokens': 7331324.0, 'completion_length': 99.340625, 'rewards/combined_reward_func': 0.6352740336209536, 'reward': 0.6352740336209536, 'reward_std': 0.038073176372563466, 'kl': 0.020635014398067142, 'clip_ratio': 0.0, 'epoch': 3.04}
Step 770: {'loss': -0.0148, 'grad_norm': 0.22488729655742645, 'learning_rate': 1.1574736948340163e-05, 'num_tokens': 7429212.0, 'completion_length': 99.8, 'rewards/combined_reward_func': 0.607759565860033, 'reward': 0.607759565860033, 'reward_std': 0.08864488755352795, 'kl': 0.01794062868502806, 'clip_ratio': 0.0, 'epoch': 3.08}
{'loss': -0.0148, 'grad_norm': 0.22488729655742645, 'learning_rate': 1.1574736948340163e-05, 'num_tokens': 7429212.0, 'completion_length': 99.8, 'rewards/combined_reward_func': 0.607759565860033, 'reward': 0.607759565860033, 'reward_std': 0.08864488755352795, 'kl': 0.01794062868502806, 'clip_ratio': 0.0, 'epoch': 3.08}
Step 780: {'loss': 0.0009, 'grad_norm': 0.01184244267642498, 'learning_rate': 1.1168313630963145e-05, 'num_tokens': 7525291.0, 'completion_length': 88.49375, 'rewards/combined_reward_func': 0.6561212211847305, 'reward': 0.6561212211847305, 'reward_std': 0.0617114937864244, 'kl': 0.016462904085346963, 'clip_ratio': 0.0, 'epoch': 3.12}
{'loss': 0.0009, 'grad_norm': 0.01184244267642498, 'learning_rate': 1.1168313630963145e-05, 'num_tokens': 7525291.0, 'completion_length': 88.49375, 'rewards/combined_reward_func': 0.6561212211847305, 'reward': 0.6561212211847305, 'reward_std': 0.0617114937864244, 'kl': 0.016462904085346963, 'clip_ratio': 0.0, 'epoch': 3.12}
Step 790: {'loss': -0.0212, 'grad_norm': 0.17011450231075287, 'learning_rate': 1.0764878147356852e-05, 'num_tokens': 7621208.0, 'completion_length': 87.48125, 'rewards/combined_reward_func': 0.6253950461745262, 'reward': 0.6253950461745262, 'reward_std': 0.0744468342512846, 'kl': 0.013984467273257906, 'clip_ratio': 0.0, 'epoch': 3.16}
{'loss': -0.0212, 'grad_norm': 0.17011450231075287, 'learning_rate': 1.0764878147356852e-05, 'num_tokens': 7621208.0, 'completion_length': 87.48125, 'rewards/combined_reward_func': 0.6253950461745262, 'reward': 0.6253950461745262, 'reward_std': 0.0744468342512846, 'kl': 0.013984467273257906, 'clip_ratio': 0.0, 'epoch': 3.16}
Step 800: {'loss': 0.0132, 'grad_norm': 0.2326195389032364, 'learning_rate': 1.036474508437579e-05, 'epoch': 3.2}
{'loss': 0.0132, 'grad_norm': 0.2326195389032364, 'learning_rate': 1.036474508437579e-05, 'epoch': 3.2}
Step 800: {'eval_loss': -0.003165504662320018, 'eval_runtime': 671.4958, 'eval_samples_per_second': 0.298, 'eval_steps_per_second': 0.074, 'eval_num_tokens': 7716872.0, 'eval_completion_length': 88.19625, 'eval_rewards/combined_reward_func': 0.6304260532557965, 'eval_reward': 0.6304260532557965, 'eval_reward_std': 0.08000247531570494, 'eval_kl': 0.018439034952025396, 'eval_clip_ratio': 0.0, 'epoch': 3.2}
{'eval_loss': -0.003165504662320018, 'eval_runtime': 671.4958, 'eval_samples_per_second': 0.298, 'eval_steps_per_second': 0.074, 'eval_num_tokens': 7716872.0, 'eval_completion_length': 88.19625, 'eval_rewards/combined_reward_func': 0.6304260532557965, 'eval_reward': 0.6304260532557965, 'eval_reward_std': 0.08000247531570494, 'eval_kl': 0.018439034952025396, 'eval_clip_ratio': 0.0, 'epoch': 3.2}
Step 810: {'loss': -0.0311, 'grad_norm': 0.1249484196305275, 'learning_rate': 9.968226453746177e-06, 'num_tokens': 7813544.0, 'completion_length': 89.05, 'rewards/combined_reward_func': 0.632242988049984, 'reward': 0.632242988049984, 'reward_std': 0.09128812816925347, 'kl': 0.01600686257952475, 'clip_ratio': 0.0, 'epoch': 3.24}
{'loss': -0.0311, 'grad_norm': 0.1249484196305275, 'learning_rate': 9.968226453746177e-06, 'num_tokens': 7813544.0, 'completion_length': 89.05, 'rewards/combined_reward_func': 0.632242988049984, 'reward': 0.632242988049984, 'reward_std': 0.09128812816925347, 'kl': 0.01600686257952475, 'clip_ratio': 0.0, 'epoch': 3.24}
Step 820: {'loss': -0.0057, 'grad_norm': 0.18619932234287262, 'learning_rate': 9.575631448768618e-06, 'num_tokens': 7910296.0, 'completion_length': 92.7, 'rewards/combined_reward_func': 0.6102332055568696, 'reward': 0.6102332055568696, 'reward_std': 0.1042077316902578, 'kl': 0.022010866337950574, 'clip_ratio': 0.0, 'epoch': 3.2800000000000002}
{'loss': -0.0057, 'grad_norm': 0.18619932234287262, 'learning_rate': 9.575631448768618e-06, 'num_tokens': 7910296.0, 'completion_length': 92.7, 'rewards/combined_reward_func': 0.6102332055568696, 'reward': 0.6102332055568696, 'reward_std': 0.1042077316902578, 'kl': 0.022010866337950574, 'clip_ratio': 0.0, 'epoch': 3.28}
Step 830: {'loss': 0.0096, 'grad_norm': 0.13534663617610931, 'learning_rate': 9.187266203218457e-06, 'num_tokens': 8007447.0, 'completion_length': 95.19375, 'rewards/combined_reward_func': 0.6461462393403054, 'reward': 0.6461462393403054, 'reward_std': 0.12807315753307194, 'kl': 0.019450197544210825, 'clip_ratio': 0.0, 'epoch': 3.32}
{'loss': 0.0096, 'grad_norm': 0.13534663617610931, 'learning_rate': 9.187266203218457e-06, 'num_tokens': 8007447.0, 'completion_length': 95.19375, 'rewards/combined_reward_func': 0.6461462393403054, 'reward': 0.6461462393403054, 'reward_std': 0.12807315753307194, 'kl': 0.019450197544210825, 'clip_ratio': 0.0, 'epoch': 3.32}
Step 840: {'loss': -0.0365, 'grad_norm': 0.25226759910583496, 'learning_rate': 8.803433552631876e-06, 'num_tokens': 8104192.0, 'completion_length': 92.65625, 'rewards/combined_reward_func': 0.6199618399143219, 'reward': 0.6199618399143219, 'reward_std': 0.1502029186114669, 'kl': 0.019328462581324857, 'clip_ratio': 0.0, 'epoch': 3.36}
{'loss': -0.0365, 'grad_norm': 0.25226759910583496, 'learning_rate': 8.803433552631876e-06, 'num_tokens': 8104192.0, 'completion_length': 92.65625, 'rewards/combined_reward_func': 0.6199618399143219, 'reward': 0.6199618399143219, 'reward_std': 0.1502029186114669, 'kl': 0.019328462581324857, 'clip_ratio': 0.0, 'epoch': 3.36}
Step 850: {'loss': -0.0023, 'grad_norm': 0.2639930546283722, 'learning_rate': 8.424432798163838e-06, 'epoch': 3.4}
{'loss': -0.0023, 'grad_norm': 0.2639930546283722, 'learning_rate': 8.424432798163838e-06, 'epoch': 3.4}
Step 850: {'eval_loss': 0.006933643016964197, 'eval_runtime': 682.7128, 'eval_samples_per_second': 0.293, 'eval_steps_per_second': 0.073, 'eval_num_tokens': 8199860.0, 'eval_completion_length': 89.14625, 'eval_rewards/combined_reward_func': 0.6556518496572972, 'eval_reward': 0.6556518496572972, 'eval_reward_std': 0.08328549488913267, 'eval_kl': 0.009149043828219874, 'eval_clip_ratio': 0.0, 'epoch': 3.4}
{'eval_loss': 0.006933643016964197, 'eval_runtime': 682.7128, 'eval_samples_per_second': 0.293, 'eval_steps_per_second': 0.073, 'eval_num_tokens': 8199860.0, 'eval_completion_length': 89.14625, 'eval_rewards/combined_reward_func': 0.6556518496572972, 'eval_reward': 0.6556518496572972, 'eval_reward_std': 0.08328549488913267, 'eval_kl': 0.009149043828219874, 'eval_clip_ratio': 0.0, 'epoch': 3.4}
Step 860: {'loss': -0.0172, 'grad_norm': 0.19738218188285828, 'learning_rate': 8.050559473202078e-06, 'num_tokens': 8296602.0, 'completion_length': 89.28125, 'rewards/combined_reward_func': 0.6283748008310794, 'reward': 0.6283748008310794, 'reward_std': 0.09420122152077966, 'kl': 0.010207986322711804, 'clip_ratio': 0.0, 'epoch': 3.44}
{'loss': -0.0172, 'grad_norm': 0.19738218188285828, 'learning_rate': 8.050559473202078e-06, 'num_tokens': 8296602.0, 'completion_length': 89.28125, 'rewards/combined_reward_func': 0.6283748008310794, 'reward': 0.6283748008310794, 'reward_std': 0.09420122152077966, 'kl': 0.010207986322711804, 'clip_ratio': 0.0, 'epoch': 3.44}
Step 870: {'loss': -0.0033, 'grad_norm': 0.12016753107309341, 'learning_rate': 7.682105112919007e-06, 'num_tokens': 8392373.0, 'completion_length': 86.56875, 'rewards/combined_reward_func': 0.6463606886565685, 'reward': 0.6463606886565685, 'reward_std': 0.10325463400222361, 'kl': 0.011147383082425221, 'clip_ratio': 0.0, 'epoch': 3.48}
{'loss': -0.0033, 'grad_norm': 0.12016753107309341, 'learning_rate': 7.682105112919007e-06, 'num_tokens': 8392373.0, 'completion_length': 86.56875, 'rewards/combined_reward_func': 0.6463606886565685, 'reward': 0.6463606886565685, 'reward_std': 0.10325463400222361, 'kl': 0.011147383082425221, 'clip_ratio': 0.0, 'epoch': 3.48}
Step 880: {'loss': 0.0047, 'grad_norm': 0.09549775719642639, 'learning_rate': 7.319357026941429e-06, 'num_tokens': 8488781.0, 'completion_length': 90.55, 'rewards/combined_reward_func': 0.6612723842263222, 'reward': 0.6612723842263222, 'reward_std': 0.08406762494705618, 'kl': 0.016983000825712226, 'clip_ratio': 0.0, 'epoch': 3.52}
{'loss': 0.0047, 'grad_norm': 0.09549775719642639, 'learning_rate': 7.319357026941429e-06, 'num_tokens': 8488781.0, 'completion_length': 90.55, 'rewards/combined_reward_func': 0.6612723842263222, 'reward': 0.6612723842263222, 'reward_std': 0.08406762494705618, 'kl': 0.016983000825712226, 'clip_ratio': 0.0, 'epoch': 3.52}
Step 890: {'loss': -0.0092, 'grad_norm': 0.047589968889951706, 'learning_rate': 6.962598075315047e-06, 'num_tokens': 8585060.0, 'completion_length': 89.74375, 'rewards/combined_reward_func': 0.6248626127839089, 'reward': 0.6248626127839089, 'reward_std': 0.05304563762620092, 'kl': 0.015287969372002408, 'clip_ratio': 0.0, 'epoch': 3.56}
{'loss': -0.0092, 'grad_norm': 0.047589968889951706, 'learning_rate': 6.962598075315047e-06, 'num_tokens': 8585060.0, 'completion_length': 89.74375, 'rewards/combined_reward_func': 0.6248626127839089, 'reward': 0.6248626127839089, 'reward_std': 0.05304563762620092, 'kl': 0.015287969372002408, 'clip_ratio': 0.0, 'epoch': 3.56}
Step 900: {'loss': 0.0318, 'grad_norm': 0.18216103315353394, 'learning_rate': 6.6121064479388e-06, 'epoch': 3.6}
{'loss': 0.0318, 'grad_norm': 0.18216103315353394, 'learning_rate': 6.6121064479388e-06, 'epoch': 3.6}
Step 900: {'eval_loss': -0.012683234177529812, 'eval_runtime': 623.9176, 'eval_samples_per_second': 0.321, 'eval_steps_per_second': 0.08, 'eval_num_tokens': 8681226.0, 'eval_completion_length': 81.96, 'eval_rewards/combined_reward_func': 0.6415880480408669, 'eval_reward': 0.6415880480408669, 'eval_reward_std': 0.0892061444511637, 'eval_kl': 0.02027242549593211, 'eval_clip_ratio': 0.0, 'epoch': 3.6}
{'eval_loss': -0.012683234177529812, 'eval_runtime': 623.9176, 'eval_samples_per_second': 0.321, 'eval_steps_per_second': 0.08, 'eval_num_tokens': 8681226.0, 'eval_completion_length': 81.96, 'eval_rewards/combined_reward_func': 0.6415880480408669, 'eval_reward': 0.6415880480408669, 'eval_reward_std': 0.0892061444511637, 'eval_kl': 0.02027242549593211, 'eval_clip_ratio': 0.0, 'epoch': 3.6}
Step 910: {'loss': -0.0216, 'grad_norm': 0.21361371874809265, 'learning_rate': 6.26815544764066e-06, 'num_tokens': 8776003.0, 'completion_length': 84.696875, 'rewards/combined_reward_func': 0.6381731297820806, 'reward': 0.6381731297820806, 'reward_std': 0.09988006348721683, 'kl': 0.016200133944948902, 'clip_ratio': 0.0, 'epoch': 3.64}
{'loss': -0.0216, 'grad_norm': 0.21361371874809265, 'learning_rate': 6.26815544764066e-06, 'num_tokens': 8776003.0, 'completion_length': 84.696875, 'rewards/combined_reward_func': 0.6381731297820806, 'reward': 0.6381731297820806, 'reward_std': 0.09988006348721683, 'kl': 0.016200133944948902, 'clip_ratio': 0.0, 'epoch': 3.64}
Step 920: {'loss': -0.0066, 'grad_norm': 0.2571307420730591, 'learning_rate': 5.931013277064377e-06, 'num_tokens': 8871507.0, 'completion_length': 84.9, 'rewards/combined_reward_func': 0.6484889924526215, 'reward': 0.6484889924526215, 'reward_std': 0.10778944836929441, 'kl': 0.014048087410628796, 'clip_ratio': 0.0, 'epoch': 3.68}
{'loss': -0.0066, 'grad_norm': 0.2571307420730591, 'learning_rate': 5.931013277064377e-06, 'num_tokens': 8871507.0, 'completion_length': 84.9, 'rewards/combined_reward_func': 0.6484889924526215, 'reward': 0.6484889924526215, 'reward_std': 0.10778944836929441, 'kl': 0.014048087410628796, 'clip_ratio': 0.0, 'epoch': 3.68}
Step 930: {'loss': 0.0057, 'grad_norm': 0.1279553323984146, 'learning_rate': 5.600942829533097e-06, 'num_tokens': 8967016.0, 'completion_length': 84.93125, 'rewards/combined_reward_func': 0.6451457843184472, 'reward': 0.6451457843184472, 'reward_std': 0.058179714763537047, 'kl': 0.008088990802934858, 'clip_ratio': 0.0, 'epoch': 3.7199999999999998}
{'loss': 0.0057, 'grad_norm': 0.1279553323984146, 'learning_rate': 5.600942829533097e-06, 'num_tokens': 8967016.0, 'completion_length': 84.93125, 'rewards/combined_reward_func': 0.6451457843184472, 'reward': 0.6451457843184472, 'reward_std': 0.058179714763537047, 'kl': 0.008088990802934858, 'clip_ratio': 0.0, 'epoch': 3.72}
Step 940: {'loss': -0.0098, 'grad_norm': 0.10752798616886139, 'learning_rate': 5.2782014840530366e-06, 'num_tokens': 9062921.0, 'completion_length': 87.40625, 'rewards/combined_reward_func': 0.6588903479278088, 'reward': 0.6588903479278088, 'reward_std': 0.0673684670124203, 'kl': 0.00986012964931433, 'clip_ratio': 0.0, 'epoch': 3.76}
{'loss': -0.0098, 'grad_norm': 0.10752798616886139, 'learning_rate': 5.2782014840530366e-06, 'num_tokens': 9062921.0, 'completion_length': 87.40625, 'rewards/combined_reward_func': 0.6588903479278088, 'reward': 0.6588903479278088, 'reward_std': 0.0673684670124203, 'kl': 0.00986012964931433, 'clip_ratio': 0.0, 'epoch': 3.76}
Step 950: {'loss': -0.0186, 'grad_norm': 0.2698511481285095, 'learning_rate': 4.963040904617131e-06, 'epoch': 3.8}
{'loss': -0.0186, 'grad_norm': 0.2698511481285095, 'learning_rate': 4.963040904617131e-06, 'epoch': 3.8}
Step 950: {'eval_loss': -0.006259892135858536, 'eval_runtime': 665.6807, 'eval_samples_per_second': 0.3, 'eval_steps_per_second': 0.075, 'eval_num_tokens': 9159205.0, 'eval_completion_length': 85.07625, 'eval_rewards/combined_reward_func': 0.6560807694494725, 'eval_reward': 0.6560807694494725, 'eval_reward_std': 0.0851642864663154, 'eval_kl': 0.011731930829701014, 'eval_clip_ratio': 0.0, 'epoch': 3.8}
{'eval_loss': -0.006259892135858536, 'eval_runtime': 665.6807, 'eval_samples_per_second': 0.3, 'eval_steps_per_second': 0.075, 'eval_num_tokens': 9159205.0, 'eval_completion_length': 85.07625, 'eval_rewards/combined_reward_func': 0.6560807694494725, 'eval_reward': 0.6560807694494725, 'eval_reward_std': 0.0851642864663154, 'eval_kl': 0.011731930829701014, 'eval_clip_ratio': 0.0, 'epoch': 3.8}
Step 960: {'loss': 0.0255, 'grad_norm': 0.16274696588516235, 'learning_rate': 4.655706843964953e-06, 'num_tokens': 9253857.0, 'completion_length': 84.675, 'rewards/combined_reward_func': 0.640521639212966, 'reward': 0.640521639212966, 'reward_std': 0.08695212883176281, 'kl': 0.017946923444469576, 'clip_ratio': 0.0, 'epoch': 3.84}
{'loss': 0.0255, 'grad_norm': 0.16274696588516235, 'learning_rate': 4.655706843964953e-06, 'num_tokens': 9253857.0, 'completion_length': 84.675, 'rewards/combined_reward_func': 0.640521639212966, 'reward': 0.640521639212966, 'reward_std': 0.08695212883176281, 'kl': 0.017946923444469576, 'clip_ratio': 0.0, 'epoch': 3.84}
Step 970: {'loss': 0.0049, 'grad_norm': 0.08442547172307968, 'learning_rate': 4.356438951952189e-06, 'num_tokens': 9349702.0, 'completion_length': 87.03125, 'rewards/combined_reward_func': 0.6363171599805355, 'reward': 0.6363171599805355, 'reward_std': 0.07817451655864716, 'kl': 0.01893140080355806, 'clip_ratio': 0.0, 'epoch': 3.88}
{'loss': 0.0049, 'grad_norm': 0.08442547172307968, 'learning_rate': 4.356438951952189e-06, 'num_tokens': 9349702.0, 'completion_length': 87.03125, 'rewards/combined_reward_func': 0.6363171599805355, 'reward': 0.6363171599805355, 'reward_std': 0.07817451655864716, 'kl': 0.01893140080355806, 'clip_ratio': 0.0, 'epoch': 3.88}
Step 980: {'loss': -0.0171, 'grad_norm': 0.1468217521905899, 'learning_rate': 4.06547058867883e-06, 'num_tokens': 9444968.0, 'completion_length': 83.4125, 'rewards/combined_reward_func': 0.6725767411291599, 'reward': 0.6725767411291599, 'reward_std': 0.08917466839775443, 'kl': 0.01280522284359904, 'clip_ratio': 0.0, 'epoch': 3.92}
{'loss': -0.0171, 'grad_norm': 0.1468217521905899, 'learning_rate': 4.06547058867883e-06, 'num_tokens': 9444968.0, 'completion_length': 83.4125, 'rewards/combined_reward_func': 0.6725767411291599, 'reward': 0.6725767411291599, 'reward_std': 0.08917466839775443, 'kl': 0.01280522284359904, 'clip_ratio': 0.0, 'epoch': 3.92}
Step 990: {'loss': -0.0061, 'grad_norm': 0.1532052755355835, 'learning_rate': 3.783028642522024e-06, 'num_tokens': 9540946.0, 'completion_length': 87.8625, 'rewards/combined_reward_func': 0.6486287713050842, 'reward': 0.6486287713050842, 'reward_std': 0.07788846418261527, 'kl': 0.01365463478723541, 'clip_ratio': 0.0, 'epoch': 3.96}
{'loss': -0.0061, 'grad_norm': 0.1532052755355835, 'learning_rate': 3.783028642522024e-06, 'num_tokens': 9540946.0, 'completion_length': 87.8625, 'rewards/combined_reward_func': 0.6486287713050842, 'reward': 0.6486287713050842, 'reward_std': 0.07788846418261527, 'kl': 0.01365463478723541, 'clip_ratio': 0.0, 'epoch': 3.96}
Step 1000: {'loss': -0.0075, 'grad_norm': 0.14145499467849731, 'learning_rate': 3.5093333532153316e-06, 'epoch': 4.0}
{'loss': -0.0075, 'grad_norm': 0.14145499467849731, 'learning_rate': 3.5093333532153316e-06, 'epoch': 4.0}
Step 1000: {'eval_loss': -0.015056988224387169, 'eval_runtime': 596.4254, 'eval_samples_per_second': 0.335, 'eval_steps_per_second': 0.084, 'eval_num_tokens': 9636266.0, 'eval_completion_length': 80.5175, 'eval_rewards/combined_reward_func': 0.6534621426463127, 'eval_reward': 0.6534621426463127, 'eval_reward_std': 0.08739387524314225, 'eval_kl': 0.018473085504956542, 'eval_clip_ratio': 0.0, 'epoch': 4.0}
{'eval_loss': -0.015056988224387169, 'eval_runtime': 596.4254, 'eval_samples_per_second': 0.335, 'eval_steps_per_second': 0.084, 'eval_num_tokens': 9636266.0, 'eval_completion_length': 80.5175, 'eval_rewards/combined_reward_func': 0.6534621426463127, 'eval_reward': 0.6534621426463127, 'eval_reward_std': 0.08739387524314225, 'eval_kl': 0.018473085504956542, 'eval_clip_ratio': 0.0, 'epoch': 4.0}
Step 1010: {'loss': -0.0118, 'grad_norm': 0.3068872094154358, 'learning_rate': 3.244598140112404e-06, 'num_tokens': 9730577.0, 'completion_length': 80.596875, 'rewards/combined_reward_func': 0.6369325898587703, 'reward': 0.6369325898587703, 'reward_std': 0.08943478686269372, 'kl': 0.023701191425061552, 'clip_ratio': 0.0, 'epoch': 4.04}
{'loss': -0.0118, 'grad_norm': 0.3068872094154358, 'learning_rate': 3.244598140112404e-06, 'num_tokens': 9730577.0, 'completion_length': 80.596875, 'rewards/combined_reward_func': 0.6369325898587703, 'reward': 0.6369325898587703, 'reward_std': 0.08943478686269372, 'kl': 0.023701191425061552, 'clip_ratio': 0.0, 'epoch': 4.04}
Step 1020: {'loss': 0.0008, 'grad_norm': 0.30442482233047485, 'learning_rate': 2.9890294357689994e-06, 'num_tokens': 9824911.0, 'completion_length': 77.5875, 'rewards/combined_reward_func': 0.6608280308544636, 'reward': 0.6608280308544636, 'reward_std': 0.10266762655228376, 'kl': 0.016113664727890865, 'clip_ratio': 0.0, 'epoch': 4.08}
{'loss': 0.0008, 'grad_norm': 0.30442482233047485, 'learning_rate': 2.9890294357689994e-06, 'num_tokens': 9824911.0, 'completion_length': 77.5875, 'rewards/combined_reward_func': 0.6608280308544636, 'reward': 0.6608280308544636, 'reward_std': 0.10266762655228376, 'kl': 0.016113664727890865, 'clip_ratio': 0.0, 'epoch': 4.08}
Step 1030: {'loss': 0.0072, 'grad_norm': 0.20911723375320435, 'learning_rate': 2.7428265249730726e-06, 'num_tokens': 9920127.0, 'completion_length': 83.1, 'rewards/combined_reward_func': 0.6660845279693604, 'reward': 0.6660845279693604, 'reward_std': 0.06472027325071394, 'kl': 0.009193637329735794, 'clip_ratio': 0.0, 'epoch': 4.12}
{'loss': 0.0072, 'grad_norm': 0.20911723375320435, 'learning_rate': 2.7428265249730726e-06, 'num_tokens': 9920127.0, 'completion_length': 83.1, 'rewards/combined_reward_func': 0.6660845279693604, 'reward': 0.6660845279693604, 'reward_std': 0.06472027325071394, 'kl': 0.009193637329735794, 'clip_ratio': 0.0, 'epoch': 4.12}
Step 1040: {'loss': -0.0098, 'grad_norm': 0.19846419990062714, 'learning_rate': 2.5061813893485087e-06, 'num_tokens': 10015320.0, 'completion_length': 82.95625, 'rewards/combined_reward_func': 0.6330464407801628, 'reward': 0.6330464407801628, 'reward_std': 0.07993211573921144, 'kl': 0.018792881406261584, 'clip_ratio': 0.0, 'epoch': 4.16}
{'loss': -0.0098, 'grad_norm': 0.19846419990062714, 'learning_rate': 2.5061813893485087e-06, 'num_tokens': 10015320.0, 'completion_length': 82.95625, 'rewards/combined_reward_func': 0.6330464407801628, 'reward': 0.6330464407801628, 'reward_std': 0.07993211573921144, 'kl': 0.018792881406261584, 'clip_ratio': 0.0, 'epoch': 4.16}
Step 1050: {'loss': 0.0141, 'grad_norm': 0.11309938132762909, 'learning_rate': 2.279278557653611e-06, 'epoch': 4.2}
{'loss': 0.0141, 'grad_norm': 0.11309938132762909, 'learning_rate': 2.279278557653611e-06, 'epoch': 4.2}