-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Tensorboard
After you start training, you are able to inspect its progress by using Tensorboard. (Note that you should separately install Tensorboard in advance.)
If we assume that your output path is /my/training/output/path in config.json, then you can run Tensorboard by the below command in your terminal. After that, you can reach Tensorboard from localhost:6006 in your browser.
tensorboard --logdir=my_run:/my/training/output/path
TTS provides quite extensive Tensorboard logging.
- Training and validation loss plots.
- Attention figures.
- Training, validation and testing audio samples.
- Layer wise weight and grad histograms.
You can check here to see how a healthy training looks like.
Under '''SCALARS''' tab, you find plots showing train/validation losses, gradient-norm, epoch and iteration times. [TODO]
Under '''IMAGES''' tab, you see train/validation/test attention plots. Train plots are updated after each checkpoint and validation/test plots are updated after each epoch.
It is important to see a diagonal alignment here for a salient model. Sometimes, your model might be optimizing loss values well but unless you have a good attention alignment, your results would be unintelligible. [TODO]
Under '''AUDIO''' tab, there are sample synthesis results of the model. These samples are generated to hear the model performance on the fly. However, it is important to notice that training/validation results are generated with teacher-forcing. Therefore, they are always better then test results. However, test results are obtained by using the exact same setting in inference time. Therefore, they are the best indicator for the model performance.
[TODO]
If you enable tb_model_param_stats in config.json, you see weight and gradient stats per layer under '''SCALARS''', '''DISTRIBUTIONS''' and '''HISTOGRAMS''' tabs.
Under '''SCALARS''' tab, there are plots depicting max/min/mean/std of layer weights.
Under '''DISTRIBUTIONS''', you are able to see the weight and gradient distribution through time.
Under '''HISTOGRAMS''', layer weight and grad values are shown in a nice incremental way.
These stats are useful to detect any problem of your model, especially, if you are after a new dataset or a new architectural change. So you can find the flawed layer by inspecting figures.
[TODO]