Explore LR discrepancy between instructlab-training and mini trainer #18
Description
It seems like learning rates of 2e-5 in mini-trainer correspond to a learning rate of 5e-6 in instructlab-training. We need to explore why.
{{ message }}
It seems like learning rates of 2e-5 in mini-trainer correspond to a learning rate of 5e-6 in instructlab-training. We need to explore why.