I just trained an MMLV model on deterministic captions for 300 epochs, though I've seen some issues similar to what I'm about to describe when training for 500 epochs as well. Looking at these results partially inspired issue #244 but there is a bigger issue here. This is the training loss vs validation loss plot:
This looks like overfitting. The validation loss dips down to 0.32 then ticks back up to 0.35 while the training loss keeps going down. This doesn't really happen with MM-Full or MM-Simple, and definitely not with Mario, though in the Mario paper we did differentiate the "best" model from the final model in terms of caption score, though I think the code in train_diffusion.py is now set up to use validation loss when training with caption_keys.
So, first try training with early stopping on and see what happens. When does training stop and how do the results look? For the sake of comparison, I've usually preferred to train for a fixed number of epochs and then use the best model encountered over that period (even if it was not the final model), but I'm honestly not fully sure how the code handles that any more (easy to forget with time and complexity) so look into that. If training for 300 epochs actually finds a better model at epoch 200, then not only should we save the best model instead, but some output in the model directory should make it very clear/explicit that this is what happened, and report which epoch the model came from.
However, more enticing is the idea that we could get the validation loss lower than we are seeing. This is where tweaking the learning rate scheduler could have an impact. There are so many things we could try, and each time we try something, it takes time, so I'm not sure how much effort to dedicate to this, but it is worth looking into.
I just trained an MMLV model on deterministic captions for 300 epochs, though I've seen some issues similar to what I'm about to describe when training for 500 epochs as well. Looking at these results partially inspired issue #244 but there is a bigger issue here. This is the training loss vs validation loss plot:
This looks like overfitting. The validation loss dips down to 0.32 then ticks back up to 0.35 while the training loss keeps going down. This doesn't really happen with MM-Full or MM-Simple, and definitely not with Mario, though in the Mario paper we did differentiate the "best" model from the final model in terms of caption score, though I think the code in train_diffusion.py is now set up to use validation loss when training with caption_keys.
So, first try training with early stopping on and see what happens. When does training stop and how do the results look? For the sake of comparison, I've usually preferred to train for a fixed number of epochs and then use the best model encountered over that period (even if it was not the final model), but I'm honestly not fully sure how the code handles that any more (easy to forget with time and complexity) so look into that. If training for 300 epochs actually finds a better model at epoch 200, then not only should we save the best model instead, but some output in the model directory should make it very clear/explicit that this is what happened, and report which epoch the model came from.
However, more enticing is the idea that we could get the validation loss lower than we are seeing. This is where tweaking the learning rate scheduler could have an impact. There are so many things we could try, and each time we try something, it takes time, so I'm not sure how much effort to dedicate to this, but it is worth looking into.