I notice that I create a model using tensorflow.keras.Sequential(), save it and the file size is around 5 MiB, but after I call model.fit(..), the file size is increased to 17 MiB. I copied the model to reduce the filesize and see that the accuracy is the same.

My question is, what exactly is the content of extra 12 MiB that fit() produces? How can I access such content? If I remove those extra 12 MiB, could it affect prediction accuracy or any weird side-effect?

See my experiment code here: https://nbviewer.jupyter.org/github/off99555/TensorFlowExperiments/blob/master/test-save-keras-model.ipynb

  • $\begingroup$ maybe batch normalisation info ? :D hm or dkn maybe some history is stored $\endgroup$ – user8426627 Jul 16 '19 at 12:53
  • $\begingroup$ @user8426627 See my answer. $\endgroup$ – off99555 Jul 16 '19 at 13:56
  • $\begingroup$ model.save also saves the optimiser's state. Try model.save_weights $\endgroup$ – Abhijit Balaji Jul 16 '19 at 14:01
  • $\begingroup$ model.save_weights() doesn't save the model architecture. So it's not convenient for loading the model again. $\endgroup$ – off99555 Jul 16 '19 at 14:08

The answer is that it's the size of the Adam optimizer state. When I change the optimizer to SGD (the vanilla optimizer), the size is not big anymore. As far as I know, the Adam optimizer maintains gradients information of previous training iterations. And the gradient size can be as big as the model size. That's why it causes the file size to be so big.

With this in mind, when you save your model please make sure to set include_optimizer=False if you seem to use an optimizer that maintains big state similar to Adam.

Beware though, it means that you cannot load the model and continue training it again, it should only be used for inference.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.