0
$\begingroup$

My CNN tensorflow model reports 100% validation accuracy within 2 epochs. But it incorrectly predicts on single new images. (It is multiclass problem. I have 3 classes). How to resolve this? Can you please help me understand these epoch results?

I have 1,000 images per class that are representative of my testing data. How can validation accuracy reach 1.00 in just the first epoch when I have a dataset of 3,000 images in total, equal amount per class? (I would expect this to start at around 33% percent -- 1/ 3 classes.)

I understand overfitting can be a problem. I've added a dropout layer to try to solve this potential problem. From this questionWhat to do if CNN cannot overfit a training set on adding dropout? I learned that a "model is over-fitting if during training your training loss continues to decrease but (in the later epochs) your validation loss begins to increase. That means the model can not generalize well to images it has not previously encountered." I don't believe my model is overfitting based on this description. (My model reports both high training and high validation accuracy. If my model was overfitting I'd expect high training accuracy and low validation accuracy.)

My model:

def model():
  model_input = tf.keras.layers.Input(shape=(h, w, 3)) 
  x = tf.keras.layers.Rescaling(rescale_factor)(model_input) 
  x = tf.keras.layers.Conv2D(16, 3, activation='relu',padding='same')(x)
  x = tf.keras.layers.Dropout(.5)(x)
  x = tf.keras.layers.MaxPooling2D()(x) 
  x = tf.keras.layers.Flatten()(x)
  x = tf.keras.layers.Dense(128, activation='relu')(x)
  outputs = tf.keras.layers.Dense(num_classes, activation = 'softmax')(x)

Epoch results:

Epoch 1/10
/usr/local/lib/python3.7/dist-packages/tensorflow/python/util/dispatch.py:1096: UserWarning: "`sparse_categorical_crossentropy` received `from_logits=True`, but the `output` argument was produced by a sigmoid or softmax activation and thus does not represent logits. Was this intended?"
  return dispatch_target(*args, **kwargs)
27/27 [==============================] - 13s 124ms/step - loss: 1.0004 - accuracy: 0.5953 - val_loss: 0.5053 - val_accuracy: 0.8920
Epoch 2/10
27/27 [==============================] - 1s 46ms/step - loss: 0.1368 - accuracy: 0.9825 - val_loss: 0.0126 - val_accuracy: 1.0000
Epoch 3/10
27/27 [==============================] - 1s 42ms/step - loss: 0.0020 - accuracy: 1.0000 - val_loss: 5.9116e-04 - val_accuracy: 1.0000
Epoch 4/10
27/27 [==============================] - 1s 42ms/step - loss: 3.0633e-04 - accuracy: 1.0000 - val_loss: 3.5376e-04 - val_accuracy: 1.0000
Epoch 5/10
27/27 [==============================] - 1s 42ms/step - loss: 1.7445e-04 - accuracy: 1.0000 - val_loss: 2.2319e-04 - val_accuracy: 1.0000
Epoch 6/10
27/27 [==============================] - 1s 42ms/step - loss: 1.2910e-04 - accuracy: 1.0000 - val_loss: 1.8078e-04 - val_accuracy: 1.0000
Epoch 7/10
27/27 [==============================] - 1s 42ms/step - loss: 1.0425e-04 - accuracy: 1.0000 - val_loss: 1.4247e-04 - val_accuracy: 1.0000
Epoch 8/10
27/27 [==============================] - 1s 42ms/step - loss: 8.6284e-05 - accuracy: 1.0000 - val_loss: 1.2057e-04 - val_accuracy: 1.0000
Epoch 9/10
27/27 [==============================] - 1s 42ms/step - loss: 7.0085e-05 - accuracy: 1.0000 - val_loss: 9.3485e-05 - val_accuracy: 1.0000
Epoch 10/10
27/27 [==============================] - 1s 42ms/step - loss: 5.4979e-05 - accuracy: 1.0000 - val_loss: 8.5952e-05 - val_accuracy: 1.0000

Model.fit and model.compile:

model = model()

model = tf.keras.Model(model_input, outputs)
  
 model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),
              metrics=['accuracy'])
  
hist = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=10
)

Code to predict new image:

def makePrediction(image):
  from IPython.display import display
  from PIL import Image
  from tensorflow.keras.preprocessing import image_dataset_from_directory 
  img = keras.preprocessing.image.load_img(
  image, target_size=(h, q)
  )
  img_array = keras.preprocessing.image.img_to_array(img)
  img_array = tf.expand_dims(img_array, 0) #Create a batch
 
  predicts = model.predict(img_array)
  p = class_names[np.argmax(predicts)]
  return p

Going to the "data" directory and using the folders to create a dataset. Each folder is a class label:

from keras.preprocessing import image
directory_data = "data"
tf.keras.utils.image_dataset_from_directory(
    directory_testData, labels='inferred', label_mode='int',
    class_names=None, color_mode='rgb', batch_size=32, image_size=(256,
    256), shuffle=True, seed=123, validation_split=0.2, subset="validation",
    interpolation='bilinear', follow_links=False,
    crop_to_aspect_ratio=False
)
 
tf.keras.utils.image_dataset_from_directory(directory_testData, labels='inferred')

Creating dataset and splitting it:

Train_ds code: (Output: Found 1605 files belonging to 3 classes. Using 1284 files for training.)

train_ds = tf.keras.preprocessing.image_dataset_from_directory(
  directory_data = "data",
  validation_split=0.2,
  subset="training",
  seed=123,
  image_size=(h, w),
  batch_size=batch_size)

Val_ds code: (Output: Found 1605 files belonging to 3 classes. Using 321 files for validation.)

val_ds = tf.keras.preprocessing.image_dataset_from_directory(
directory_data = "data",
  validation_split=0.2,
  subset="validation",
  seed=123,
  image_size=(h, w),
  batch_size=batch_size)
$\endgroup$
10
  • $\begingroup$ it can be data leakage. Check whether train and val images are same or not. $\endgroup$ Nov 28 '21 at 3:38
  • $\begingroup$ @PrakashDahal Thank you for your help! I tried my best to check this. I don't think they are the same, and I believe I have split correctly. I would appreciate your help to ensure that data is not leaking, as I am new to ML :) $\endgroup$
    – user4561
    Nov 28 '21 at 3:59
  • $\begingroup$ I also checked my batch size to ensure it is reasonable. (If all the data is being passed to the model in 1 or 2 epochs then it may make sense for val accuracy to be high within first 2 epochs) My batch size is 32, so the model will not see all data in the first 2 epochs. $\endgroup$
    – user4561
    Nov 28 '21 at 4:03
  • $\begingroup$ The batch size is used to control number of items used per weight update step, it does not affect the size of the epoch - an epoch is by definition a loop through all the training data. In your case you would expect around 40 weight updates per epoch. $\endgroup$ Nov 28 '21 at 10:10
  • $\begingroup$ @NeilSlater Thank you for explaining! I would appreciate any additional pointers for things to consider as I try to understand why val acciuracy is 1.00 in the first epoch and why my model then incorrectly classifies all new images. $\endgroup$
    – user4561
    Nov 28 '21 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.