the same Microsoft group has released code for natural speech 3 https://speechresearch.github.io/naturalspeech3/ Could this help?