1
$\begingroup$

I've been following a deep learning book and the current section I'm on is about convolutional neural networks. The author presents some code to create a basic CNN with about 1 million parameters, which he manages to train to 99.2% accuracy within 12 epochs on the full MNIST dataset.

His output looks like this:

Train on 60000 samples, validate on 10000 samples 
Epoch 1/12-loss:0.2800 acc:0.9147 val_loss:0.0624 val_acc:0.9794 
Epoch 2/12-loss:0.1003 acc:0.9695 val_loss:0.0422 val_acc:0.9854 
Epoch 3/12-loss:0.0697 acc:0.9789 val_loss:0.0356 val_acc:0.9880 
Epoch 4/12-loss:0.0573 acc:0.9827 val_loss:0.0282 val_acc:0.9910 
Epoch 5/12-loss:0.0478 acc:0.9854 val_loss:0.0311 val_acc:0.9901 
Epoch 6/12-loss:0.0419 acc:0.9871 val_loss:0.0279 val_acc:0.9908 
Epoch 7/12-loss:0.0397 acc:0.9883 val_loss:0.0250 val_acc:0.9914 
Epoch 8/12-loss:0.0344 acc:0.9891 val_loss:0.0288 val_acc:0.9910 
Epoch 9/12-loss:0.0329 acc:0.9895 val_loss:0.0273 val_acc:0.9916 
Epoch 10/12-loss:0.0305 acc:0.9909 val_loss:0.0296 val_acc:0.9904 
Epoch 11/12-loss:0.0291 acc:0.9911 val_loss:0.0275 val_acc:0.9920 
Epoch 12/12-loss:0.0274 acc:0.9916 val_loss:0.0245 val_acc:0.9916 
Test loss:
0.02452171179684301 Test accuracy: 0.9916

Using this code.

Running that same code on my machine, after 12 epochs, I'm barely at 0.6 accuracy. I did have to modify a couple function calls as keras has changed a couple things since the book came out. I'm going crazy trying to figure out why his is so quick. He presents the results as "this is what should happen when the code is run." I understand it had already went through almost 500 gradient descent steps by the time epoch 1 is complete, but is that really enough to reach 98% accuracy right off the bat??

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

I was able to run the code without "any" modifications on Tensorflow 2.4.0, just had to replace the imports:

import keras
from keras.datasets import mnist
...

->

import tensorflow.keras as keras
from tensorflow.keras.datasets import mnist
...

Output:

Epoch 1/12
469/469 [==============================] - 4s 7ms/step - loss: 2.2914 - accuracy: 0.1260 - val_loss: 2.2425 - val_accuracy: 0.2915
Epoch 2/12
469/469 [==============================] - 3s 7ms/step - loss: 2.2375 - accuracy: 0.2383 - val_loss: 2.1710 - val_accuracy: 0.4693
Epoch 3/12
469/469 [==============================] - 3s 7ms/step - loss: 2.1665 - accuracy: 0.3439 - val_loss: 2.0758 - val_accuracy: 0.5645
Epoch 4/12
469/469 [==============================] - 3s 7ms/step - loss: 2.0750 - accuracy: 0.4286 - val_loss: 1.9489 - val_accuracy: 0.6155
Epoch 5/12
469/469 [==============================] - 3s 7ms/step - loss: 1.9510 - accuracy: 0.4964 - val_loss: 1.7919 - val_accuracy: 0.6633
Epoch 6/12
469/469 [==============================] - 4s 8ms/step - loss: 1.8107 - accuracy: 0.5401 - val_loss: 1.6096 - val_accuracy: 0.7082
Epoch 7/12
469/469 [==============================] - 3s 7ms/step - loss: 1.6469 - accuracy: 0.5737 - val_loss: 1.4162 - val_accuracy: 0.7475
Epoch 8/12
469/469 [==============================] - 3s 7ms/step - loss: 1.4888 - accuracy: 0.6095 - val_loss: 1.2314 - val_accuracy: 0.7727
Epoch 9/12
469/469 [==============================] - 3s 7ms/step - loss: 1.3350 - accuracy: 0.6409 - val_loss: 1.0702 - val_accuracy: 0.7961
Epoch 10/12
469/469 [==============================] - 3s 7ms/step - loss: 1.2118 - accuracy: 0.6651 - val_loss: 0.9390 - val_accuracy: 0.8161
Epoch 11/12
469/469 [==============================] - 3s 7ms/step - loss: 1.1072 - accuracy: 0.6855 - val_loss: 0.8373 - val_accuracy: 0.8291
Epoch 12/12
469/469 [==============================] - 3s 7ms/step - loss: 1.0208 - accuracy: 0.7061 - val_loss: 0.7568 - val_accuracy: 0.8370

Test loss: 0.7567713856697083
Test accuracy: 0.8370000123977661

Running it 10 times (it took about 30 seconds / model on 1080 Ti):

Test loss: 0.71385968 +/- 0.0471146
Test accuracy: 0.84305001 +/- 0.00868576

So clearly these are very different results than the author got, but also significantly higher than what You got. He might have used different parameters with the Adadelta optimizer, or a different version of it.

Usually I've had the best results with Adam (default parameters, learning rate = 0.001). I tested running it 5 times with these results:

Test loss: 0.02858894 +/- 0.00151724
Test accuracy: 0.99186 +/- 0.00058172

Adam is 3 years newer than Adadelta (2014 vs. 2011).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you so much! This really clears things up. I think 0.6 was the lowest I got. My more recent ones were in the same range as yours. It was just the instant high accuracy and low loss from his data that was really throwing me off. However, I'm able to mimic that using Adam like you said. Happy to see everything's running as its supposed to. Guess I'll stick to Adam for now. $\endgroup$
    – rad
    Jan 5, 2022 at 21:08
0
$\begingroup$

According to my experience, it is possible to reach 99%+ accuracy on MNIST within a few epochs using a simple CNN. MNIST is really an easy dataset. So, it's likely that you've broken something as you're modifying author's code.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I didn't really "modify" much. The compiler couldn't find the to_categorical() or the Adadelta() functions in the imported libraries, so I imported tensorflow itself and prefixed the function call with it. $\endgroup$
    – rad
    Jan 5, 2022 at 18:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .