Activation Offloading for Lora DPO

[Vattikondadheeraj]

Hey, I am wondering, if activation offloading is present in the lora distributed DPO or not? because I cannot find any terms related to activation offloading. If not, are there plans to release it?

[SalmanMohammadi]

Hey @Vattikondadheeraj. This should be possible - I'll take a look.

[SalmanMohammadi]

This feature should be included in our next nightly build, and our upcoming 0.5 stable release @Vattikondadheeraj! Give it a try : )