-2
$\begingroup$

I was trying to write a simple CNN in keras during a course, and I wrote one that does not learn at all, but I don't understand why.

Don't bother about the coding, first I load two images of a dog and one of a cat :

img1 = np.array(Image.open('dog1.jpg').convert('L'))
img2 = np.array(Image.open('dog2.jpg').convert('L'))
img3 = np.array(Image.open('dog3.jpg').convert('L'))
img4 = np.array(Image.open('cat1.png').convert('L').resize((225,225)))

X = np.array([img1.reshape(1,225,225),img2.reshape(1,225,225),img4.reshape(1,225,225)])
y = np.array([np.array([1,0]),np.array([1,0]),np.array([0,1])])

then I define the CNN model :

model = Sequential()
model.add(Conv2D(96, (11, 11), input_shape=(1, 225, 225), activation='relu'))
model.add(MaxPooling2D(pool_size=(3, 3),strides=2))
model.add(Flatten())
model.add(Dense(2,activation='relu'))

and then I train the CNN model :

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X,y,epochs=5)

model.predict(img3.reshape(1,1,225,225))

I always get the following for the training (even for epochs=100) :

Epoch 1/5
3/3 [==============================] - 2s 778ms/step - loss: 5.3727 - acc: 0.6667
Epoch 2/5
3/3 [==============================] - 2s 645ms/step - loss: 5.3727 - acc: 0.6667
Epoch 3/5
3/3 [==============================] - 2s 649ms/step - loss: 5.3727 - acc: 0.6667
Epoch 4/5
3/3 [==============================] - 2s 640ms/step - loss: 5.3727 - acc: 0.6667
Epoch 5/5
3/3 [==============================] - 2s 777ms/step - loss: 5.3727 - acc: 0.6667

and for the predict, sometimes I have values (always before the training) but often I have (always after the training) :

array([[nan, nan]], dtype=float32)

So my questions are : why does my CNN not learn at all ?? And why do I end up with "nan" after learning but have none before ? since it doesnt learn apparently, the predict shouldn't change ?

$\endgroup$
  • $\begingroup$ You have not normalized the data I guess... Data science.se is a better site to ask this sort of questions. $\endgroup$ – DuttaA Jan 21 at 16:30
  • $\begingroup$ Not normalized can explain the NaN part, but not why it doesn't learn at all no ? $\endgroup$ – lrosique Jan 21 at 17:33
  • $\begingroup$ Well what is the derivative of NaN? $\endgroup$ – DuttaA Jan 21 at 18:17
1
$\begingroup$

I modified your example to get it to converge. The following changes were made:

  • Use softmax as the activation of your output layer, not relu. Using relu caused nan results. The loss categorical_crossentropy expects a probability distribution, which softmax produces, not relu.
  • Normalize the image values to be between 0 and 1. Not normalizing causes the network to not learn.
  • Go deeper by adding a couple more convolutional layers and max pooling layers. Without more layers the network did not learn.
  • Use a batch size of 3. This stabilized and sped convergence but was not necessary to see the network learn.

(I also switched the code to channels last format.)

Here is the modified code:

from PIL import Image
import numpy as np
import tensorflow as tf
from tensorflow.keras import (Sequential)
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense
from pathlib import Path
import os
import requests
from io import BytesIO

DOWNLOAD = False

def save_and_resize(path, format, url):
    r = requests.get(url)
    img = Image.open(BytesIO(r.content)).resize((225, 225))
    img.save(path, format)


tmpdir = Path('~/tmp').expanduser()
os.chdir(tmpdir)

urls = [
    'https://s3.amazonaws.com/cdn-origin-etr.akc.org/wp-content/uploads/2017/11/12233437/Entlebucher-Mountain-Dog-On-White-01.jpg',
    'https://www.akc.org/wp-content/themes/akc/component-library/assets/img/welcome.jpg',
    'https://www.cesarsway.com/sites/newcesarsway/files/styles/large_article_preview/public/Natural-Dog-Law-2-To-dogs%2C-energy-is-everything.jpg?itok=Z-ujUOUr',
    'https://www.catster.com/wp-content/uploads/2018/07/Savannah-cat-long-body-shot.jpg'
]

if DOWNLOAD:
    save_and_resize(tmpdir / 'dog1.jpg', 'JPEG', urls[0])
    save_and_resize(tmpdir / 'dog2.jpg', 'JPEG', urls[1])
    save_and_resize(tmpdir / 'dog3.jpg', 'JPEG', urls[2])
    save_and_resize(tmpdir / 'cat1.png', 'PNG', urls[3])

img1 = np.array(Image.open('dog1.jpg').convert('L')) / 255
img2 = np.array(Image.open('dog2.jpg').convert('L')) / 255
img3 = np.array(Image.open('dog3.jpg').convert('L')) / 255
img4 = np.array(Image.open('cat1.png').convert('L').resize((225,225))) / 255

X = np.array([img1.reshape(225,225, 1),img2.reshape(225,225, 1),img4.reshape(225,225, 1)])
y = np.array([np.array([1,0]),np.array([1,0]),np.array([0,1])])

model = Sequential()
model.add(Conv2D(96, (11, 11), input_shape=(225, 225, 1), activation='relu'))
model.add(MaxPooling2D(pool_size=(3, 3),strides=2))
model.add(Flatten())
model.add(Dense(2, activation='softmax'))

model = Sequential()
model.add(Conv2D(96, (11, 11), input_shape=(225, 225, 1), activation='relu'))
model.add(MaxPooling2D(pool_size=(3, 3),strides=2))
for i in range(2):
    model.add(Conv2D(96, (3, 3), input_shape=(225, 225, 1), activation='relu'))
    model.add(MaxPooling2D(pool_size=(3, 3),strides=2))
model.add(Flatten())
model.add(Dense(2, activation='softmax'))


model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
print(model.summary()) # Examine how many parameters the model has

model.fit(X, y, epochs=20, batch_size=3)

print('test prediction [dog, cat]:', model.predict(img3.reshape(1,225,225,1)))

The output looks like:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 215, 215, 96)      11712     
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 107, 107, 96)      0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 105, 105, 96)      83040     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 52, 52, 96)        0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 50, 50, 96)        83040     
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 24, 24, 96)        0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 55296)             0         
_________________________________________________________________
dense_1 (Dense)              (None, 2)                 110594    
=================================================================
Total params: 288,386
Trainable params: 288,386
Non-trainable params: 0
_________________________________________________________________
None
Epoch 1/20
2019-01-21 12:32:42.524533: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-01-21 12:32:42.524871: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 4. Tune using inter_op_parallelism_threads for best performance.
3/3 [==============================] - 2s 724ms/step - loss: 0.7194 - acc: 0.3333
Epoch 2/20
3/3 [==============================] - 1s 185ms/step - loss: 1.8854 - acc: 0.6667
Epoch 3/20
3/3 [==============================] - 1s 177ms/step - loss: 0.5993 - acc: 0.6667
Epoch 4/20
3/3 [==============================] - 1s 172ms/step - loss: 0.7059 - acc: 0.3333
Epoch 5/20
3/3 [==============================] - 1s 177ms/step - loss: 0.5842 - acc: 0.6667
Epoch 6/20
3/3 [==============================] - 1s 172ms/step - loss: 0.5728 - acc: 0.6667
Epoch 7/20
3/3 [==============================] - 1s 179ms/step - loss: 0.5598 - acc: 1.0000
Epoch 8/20
3/3 [==============================] - 1s 171ms/step - loss: 0.4425 - acc: 0.6667
Epoch 9/20
3/3 [==============================] - 1s 171ms/step - loss: 0.3681 - acc: 0.6667
Epoch 10/20
3/3 [==============================] - 1s 182ms/step - loss: 0.2648 - acc: 1.0000
Epoch 11/20
3/3 [==============================] - 1s 173ms/step - loss: 0.1633 - acc: 1.0000
Epoch 12/20
3/3 [==============================] - 1s 172ms/step - loss: 0.0931 - acc: 1.0000
Epoch 13/20
3/3 [==============================] - 1s 174ms/step - loss: 0.0472 - acc: 1.0000
Epoch 14/20
3/3 [==============================] - 1s 172ms/step - loss: 0.0173 - acc: 1.0000
Epoch 15/20
3/3 [==============================] - 1s 176ms/step - loss: 0.0059 - acc: 1.0000
Epoch 16/20
3/3 [==============================] - 1s 171ms/step - loss: 0.0023 - acc: 1.0000
Epoch 17/20
3/3 [==============================] - 1s 174ms/step - loss: 7.3238e-04 - acc: 1.0000
Epoch 18/20
3/3 [==============================] - 1s 175ms/step - loss: 2.5119e-04 - acc: 1.0000
Epoch 19/20
3/3 [==============================] - 1s 206ms/step - loss: 8.4965e-05 - acc: 1.0000
Epoch 20/20
3/3 [==============================] - 1s 178ms/step - loss: 2.8770e-05 - acc: 1.0000
test prediction [dog, cat]: [[0.9975575  0.00244242]]
$\endgroup$
  • $\begingroup$ thank you very much for your help ! I can confirm it works, but I wanted the simplest CNN possible. If I go with no batch_size it still works (learn but not converge). For the last "relu", it was a mistake on my part, I didn't think crossentropy would need probabilities to learn. And if I remove your two extra layers, it learns only for the first epoch (but I guess it's okay since it learned) :p So in the end I can keep my code and just use softmax on last layer (it won't converge at all, but it will learn/work). Thanks again ! $\endgroup$ – lrosique Jan 21 at 17:55
0
$\begingroup$

Probably due to too few examples, Neural networks need more training datapoints to learn. With small data set, they are likely to over-fit, which seems to be the issue here. Try predicting on the training data, if it outputs correct labels there, then it is highly likely that your model is over-fitting on training data and thus losing the ability to generalize.

$\endgroup$
  • $\begingroup$ but how can it overfit ? weights are initialized at random by keras. Then I input three images, but by default the CNN can't represent any of them. Overfitting would induce a 0 loss/ 1 accuracy. After the forwarding phase, there is an error which is important, so the CNN should get updated. $\endgroup$ – lrosique Jan 21 at 13:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.