posts/how-to-scale/

[giscus[bot]]

posts/how-to-scale/

My colleagues and I always get excited when, every once in a while, deep learning research throws up a fun little maths problem. Our recent work on u-μP does just this, and in a reasonably systematic way, since we need to work out how to compensate for changes in scale (standard deviation) through deep learning ops. In this post and the accompanying notebook, we explore this problem.

https://graphcore-research.github.io/posts/how-to-scale/