Accuracy Not Going Above 30%

Question

I am trying to make a big classification model using the coco2017 dataset. Here is my code:

import tensorflow as tf
from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt
import IPython.display as display
from PIL import Image, ImageSequence
import os
import pathlib
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, Flatten, Dropout, MaxPooling2D
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import cv2
import datetime

gpus = tf.config.list_physical_devices('GPU')
if gpus:
  try:
    # Currently, memory growth needs to be the same across GPUs
    for gpu in gpus:
      tf.config.experimental.set_memory_growth(gpu, True)
    logical_gpus = tf.config.experimental.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
  except RuntimeError as e:
    # Memory growth must be set before GPUs have been initialized
    print(e)

epochs = 100
steps_per_epoch = 10
batch_size = 70
IMG_HEIGHT = 200
IMG_WIDTH = 200

train_dir = "Train"
test_dir = "Val"

train_image_generator = ImageDataGenerator(rescale=1. / 255)

test_image_generator = ImageDataGenerator(rescale=1. / 255)

train_data_gen = train_image_generator.flow_from_directory(batch_size=batch_size,
                                                           directory=train_dir,
                                                           shuffle=True,
                                                           target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                           class_mode='sparse')

test_data_gen = test_image_generator.flow_from_directory(batch_size=batch_size,
                                                         directory=test_dir,
                                                         shuffle=True,
                                                         target_size=(IMG_HEIGHT, IMG_WIDTH),
                                                         class_mode='sparse')

model = Sequential([
    Conv2D(265, 3, padding='same', activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),
    MaxPooling2D(),
    Conv2D(64, 3, padding='same', activation='relu'),
    MaxPooling2D(),
    Conv2D(32, 3, padding='same', activation='relu'),
    MaxPooling2D(),
    Flatten(),
    keras.layers.Dense(256, activation="relu"),
    keras.layers.Dense(128, activation="relu"),
    keras.layers.Dense(80, activation="softmax")
])

optimizer = tf.keras.optimizers.Adam(0.001)
optimizer.learning_rate.assign(0.0001)

model.compile(optimizer='adam',
              loss="sparse_categorical_crossentropy",
              metrics=['accuracy'])

model.summary()
tf.keras.utils.plot_model(model, to_file="model.png", show_shapes=True, show_layer_names=True, rankdir='TB')
checkpoint_path = "training/cp.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)

cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,
                                                 save_weights_only=True,
                                                 verbose=1)

os.system("rm -r logs")

log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))
history = model.fit(train_data_gen,steps_per_epoch=steps_per_epoch,epochs=epochs,validation_data=test_data_gen,validation_steps=10,callbacks=[cp_callback, tensorboard_callback])
model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))
model.save('model.h5', include_optimizer=True)

test_loss, test_acc = model.evaluate(test_data_gen)
print("Tested Acc: ", test_acc)
print("Tested Acc: ", test_acc*100, "%")

I have tried different optimizers like SGD, RMSProp, and ADAM. I also tried changing the configuration of the hidden layers. I also tried to change the metrics from accuracy to sparse_categorical_accuracy with no improvement. I cannot go beyond 30% accuracy. My guess is that the MaxPooling is doing something because I just added it but don't know what it means. Can somebody explain what the MaxPooling Layer does and what is stopping my neural network from gaining accuracy?

Answer 1

You have two questions in one.

1. Is it maxpool that ruins the model?

I would say no, the maxpool is a standard operation for convolution networks, it down-samples the intermediate representation to reduce the necessary computations, improve the regularization, and adds translation invariance to some degree. Originally averaging was used to downsample over few neighbor pixels, for example, 2x2 were averaged to one pixel. Then it was discovered max-pool often performs better in practice, where you took the max value out of these 2x2 pixels. The way you applied is ok in general.

2. Why the accuracy is not that great?

I see two issues here - first one is COCO dataset is not a classification dataset. It's an object detection dataset and there are many objects on the same image. I.e. there is an image with a person on a bicycle and a car behind him. Which class the model should assign - a person, a bicycle, or a car? The model can't know. To check if it's the issue try top-5 accuracy - it tells if the correct answer would be among top-5 guesses of the network. I would also recommend to watch the images and try to manually guess the class for few dozens of them, that would help to build the intuition

The second thing is that your model is not that deep and 30% accuracy is not bad, i.e. the random guess would be around 1% and your model doing x30 times better. You could try models like resnet - it's still quite fast, but should be doing noticeably better.

Answer 2

Accuracy is a good measure if our classes are evenly split, but is very misleading if we have imbalanced classes.Always use caution with accuracy. You need to know the distribution of the classes to know how to interpret the value.