Improving validation losses and accuracy for 3D CNN

Question

I have used a 3D CNN architecture, for detecting the presence of a particular promoter (MGMT), by using FLAIR brain scans. (64 slices per patient). The output is supposed to be binary (0/1).

I have gone through the pre-processing properly, and used stratification after splitting the "train" dataset into train and validation sets, (80-20 ratio). My model initialisation and training kernels look like this:

def get_model(width=128, height=128, depth=64):
"""Build a 3D convolutional neural network model."""

inputs = keras.Input((width, height, depth, 1))

x = layers.Conv3D(filters=64, kernel_size=3, activation="relu")(inputs)
x = layers.MaxPool3D(pool_size=2)(x)
x = layers.BatchNormalization()(x)

x = layers.Conv3D(filters=64, kernel_size=3, activation="relu")(x)
x = layers.MaxPool3D(pool_size=2)(x)
x = layers.BatchNormalization()(x)

x = layers.Conv3D(filters=128, kernel_size=3, activation="relu")(x)
x = layers.MaxPool3D(pool_size=2)(x)
x = layers.BatchNormalization()(x)

x = layers.Conv3D(filters=256, kernel_size=3, activation="relu")(x)
x = layers.MaxPool3D(pool_size=2)(x)
x = layers.BatchNormalization()(x)

x = layers.GlobalAveragePooling3D()(x)
x = layers.Dense(units=512, activation="relu")(x)
x = layers.Dropout(0.3)(x)

outputs = layers.Dense(units=1, activation="sigmoid")(x)

# Define the model.
model = keras.Model(inputs, outputs, name="3dcnn")
return model


# Build model.
model = get_model(width=128, height=128, depth=64)
model.summary()

Compile model:

initial_learning_rate = 0.0001
lr_schedule = keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate, decay_steps=100000, decay_rate=0.96, staircase=True
)
model.compile(
    loss="binary_crossentropy",
    optimizer=keras.optimizers.Adam(learning_rate=lr_schedule),
    metrics=["acc"],
)

# Define callbacks.
checkpoint_cb = keras.callbacks.ModelCheckpoint(
    "Brain_3d_classification.h5", save_best_only=True,monitor = 'val_acc', 
                             mode = 'max', verbose = 1
)
early_stopping_cb = keras.callbacks.EarlyStopping(monitor="val_acc", patience=20,mode = 'max', verbose = 1,
                           restore_best_weights = True)

# Train the model, doing validation at the end of each epoch
epochs = 60
model.fit(
    train_dataset,
    validation_data=valid_dataset,
    epochs=epochs,
    shuffle=True,
    verbose=2,
    callbacks=[checkpoint_cb, early_stopping_cb],
)

This is my first time ever working with a 3D CNN, and I used this keras webpage for the format:https://keras.io/examples/vision/3D_image_classification/

The (max) validation accuracy in my case was about 54%. I tried reducing the initial learning rate , and for 0.00001 I got to a max of 66.7%. For learning rates of 0.00005, 0.00002, I got max accuracy of about 60 and 62%.

Accuracy vs epoch plots for learning rates 0.0001, 0.00005,0.00002 and 0.00001:

It does seem like reducing the initial learning rate has a positive effect on accuracy, although the accuracy is still very low.

What other parameters can I tune to expect a better accuracy? And is it okay to just keep reducing the initial learning rate until we achieve a targeted accuracy?

I know this is a rather broad question, but I am quite confused as to how we should approach increasing the accuracy in the case of CNNs, (that too 3D), where there just seems to be a lot of stuff going on. Do I change something in my initialisations? Add more layers? Or change the parameters? Do I decrease or increase them? With so many things going on, I don't think trying every combination and just keep repeating the training process is an efficient idea...

Full notebook (including pre-processing steps): https://www.kaggle.com/shivamee682003/3d-image-preprocessing-17cd03/edit

Answer 1

What is the No Information Rate (NIR)? I.e. what are the percentages of positive and negative labels? Have you looked at the predictions of your model? If it's all 0's or all 1's then it probably learned nothing, other than predicting the majority class.

When it comes to architectural choices and hyperparameters, especially if you start working with NNs, then Andrej Karpathy's blog post called A Recipe for Training Neural Networks is a really good starting point. It gives a good reference on how to approach things in the beginning when you have not much intuition. Simply reducing the learning rate will not help much if your model is way too small. You may also find it useful to add ResNet-like skip connections to improve performance for very deep models (i.e. many layer).

Answer 2

Try removing the dropout before the prediction layer. I couldn't find the paper or article I read about this (will update the post once I find it), just found a Cross Validated post which does not add much information. As you are

If you are lowering the learning rate, you should also lower the batch size accordingly.

As for Batch Normalization layers, they probably should be applied after the convolutional layers.