Overfitting problem - poor performance on test data

Question

I'm facing the problem of overfitting and I can't deal with it - I tried experimenting with optimizer, but nothing seems appropriate. My model has extremely poor performance on testing data and the loss even rises. Is there anything I missed during the model architecture planning or training?

I'm working on GTSRB.

n_epochs = 100
n_train = 4000
n_test = 1000

def load_split(basePath, subset_type, n_samples):
    csvPath = basePath + '\\' + subset_type
    #intialize the list of data and labels
    data = []
    labels = []

    # load the contents of the CSV file, remove the first line (since it contains the CSV header)
    rows = open(csvPath).read().strip().split("\n")[1:n_samples + 1]
    random.shuffle(rows)

    #loop over the rows of csv file
    for (i, row) in enumerate(rows):
        #check to see if we should show a status update
        if i > 0 and i % 1000 == 0:
            print("[INFO] processed {} total images".format(i))

        # split the row into components and then grab the class ID and image path
        (label, imagePath) = row.strip().split(",")[-2:]
        # derive the full path to the image file and load it
        imagePath = os.path.sep.join([basePath, imagePath])
        #print(imagePath)
        image = io.imread(imagePath)

        #resize the image to be 32x32 pixels, ignoring aspect ratio, and perform CLAHE.
        image = transform.resize(image, (32, 32))
        image = exposure.equalize_adapthist(image, clip_limit = 0.1)

        #update the list of data and labels, respectively
        data.append(image)
        labels.append(int(label))

    #convert the data and labels into numpy arrays
    data = numpy.array(data)
    labels = numpy.array(labels)

    #return a tuple of the data and labels
    return (data, labels)

print("[INFO] loading training and testing data...")
(train_images, train_labels) = load_split(DATASET_PATH, 'Train.csv', n_train)
(test_images, test_labels) = load_split(DATASET_PATH, 'Test.csv', n_test)

# Normalize pixel values within 0 and 1
train_images = train_images / 255
test_images = test_images / 255

train_labels = to_categorical(train_labels, 43)
test_labels = to_categorical(test_labels, 43)

model = keras.models.Sequential()
model.add(keras.layers.Flatten(input_shape=[32, 32, 3]))
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Dense(300, activation="relu"))
model.add(keras.layers.Dropout(0.4))
model.add(keras.layers.Dense(300, activation="relu"))
model.add(keras.layers.Dropout(0.4))
model.add(keras.layers.Dense(300, activation="relu"))
model.add(keras.layers.Dense(43, activation="softmax"))
model.summary()

#Compiling the model
model.compile(optimizer='sgd', loss='categorical_crossentropy', metrics=['accuracy'])

#Fitting the model
history= model.fit(train_images,train_labels, epochs=100, batch_size=4,validation_data=(test_images,test_labels))

Answer 1

Have you tried increasing the batch size? - try 64, 128 or 256

Using a very smaller batch size (4 in your case) may not be optimal for the model to converge to the global optima.

Answer 2

I don't think this is an overfitting problem, it looks more like the model is not learning anything, but it would be useful to also see the performance on the training data to be sure. By not specifying the learning rate in the optimizer, you are using the default learning rate of 1e-2 which seems quite high. Try passing the optimizer with

keras.optimizers.SGD(learning_rate=1e-3)

instead for a learning rate of 1e-3 (or maybe try something even lower like 1e-4). And as Benjamin Cretois has mentioned, the batch size is pretty small. If you want to use smaller batch sizes you generally need a smaller learning rate as well. You can also try using other optimizers like Adam, but I think the learning rate is mainly responsible for the bad performance.