Tensorboard

Tensorboard Logging with TTS

TTS provides quite extensive Tensorboard logging.

• Training and validation loss plots.
• Attention plots.
• Training, validation and testing audio samples.
• Layer wise weight and grad histograms.

Loss plots

Under '''SCALARS''' tab, you find plots showing train/validation losses, gradient-norm, epoch and iteration times. [TODO]

Attention plots

Under '''IMAGES''' tab, you see train/validation/test attention plots. Train plots are updated after each checkpoint and validation/test plots are updated after each epoch.

It is important to see a diagonal alignment here for a salient model. Sometimes, your model might be optimizing loss values well but unless you have a good attention alignment, your results would be unintelligible. [TODO]

Audio Samples

Under '''AUDIO''' tab, there are sample synthesis results of the model. These samples are generated to hear the model performance on the fly. However, it is important to notice that training/validation results are generated with teacher-forcing. Therefore, they are always better then test results. However, test results are obtained by using the exact same setting in inference time. Therefore, they are the best indicator for the model performance.

[TODO]

Layer-wise stats

If you enable tb_model_param_stats in config.json, you see weight and gradient stats per layer under '''SCALARS''', '''DISTRIBUTIONS''' and '''HISTOGRAMS''' tabs.

Under '''SCALARS''' tab, there are plots depicting max/min/mean/std of layer weights.

Under '''DISTRIBUTIONS''', you are able to see the weight and gradient distribution through time.

Under '''HISTOGRAMS''', layer weight and grad values are shown in a nice incremental way.

These stats are useful to detect any problem of your model, especially, if you are after a new dataset or a new architectural change. So you can find the flawed layer by inspecting figures.

[TODO]