Graph of Loss on Training Set vs Test Set Graph of Accuracy

Model used

mobilenet_model = MobileNet(input_shape=in_dim, include_top=False, pooling='avg', weights='imagenet')
mob_x = Dropout(0.75)(mobilenet_model.output)
mob_x = Dense(2, activation='sigmoid')(mob_x)

model = Model(mobilenet_model.input, mob_x)

for layer in model.layers[:50]:

for layer in model.layers[50:]:


The rest of the code

in_dim = (224,224,3)
batch_size = 64
samples_per_epoch = 1000
validation_steps = 300
nb_filters1 = 32
nb_filters2 = 64
conv1_size = 3
conv2_size = 2
pool_size = 2
epochs = 20
classes_num = 2
lr = 0.000004
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
test_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
        'output/train',  # this is the target directory
        target_size= in_dim[0:2],  # all images will be resized to 224*224
#Found 6062 images belonging to 2 classes.
validation_generator = test_datagen.flow_from_directory(
#Found 769 images belonging to 2 classes.
from keras.callbacks import EarlyStopping
#set early stopping monitor so the model stops training when it won't improve anymore
early_stopping_monitor = EarlyStopping(patience=3)
steps_per_epoch = 10
from keras import backend as K

def recall_m(y_true, y_pred):
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
        recall = true_positives / (possible_positives + K.epsilon())
        return recall

def precision_m(y_true, y_pred):
        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
        precision = true_positives / (predicted_positives + K.epsilon())
        return precision

def f1_m(y_true, y_pred):
    precision = precision_m(y_true, y_pred)
    recall = recall_m(y_true, y_pred)
    return 2*((precision*recall)/(precision+recall+K.epsilon()))
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc',f1_m,precision_m, recall_m])

history = model.fit_generator(
        steps_per_epoch=2000// batch_size ,
        validation_steps=800// batch_size,
        callbacks = [early_stopping_monitor],
test_generator = train_datagen.flow_from_directory(
loss, accuracy, f1_score, precision, recall = model.evaluate(test_generator)
print("The test set accuracy is ", accuracy)
#The test set accuracy is  0.9001349538122272

From what I have gathered from this post and this article, I understand that the validation set is much smaller with respect to the training set. I have applied augmentation to the test set due to this and that boosted test set accuracy by 1%.

Please note that the test train split is "stratified" as here is a breakdown of each individual class in test/train/validation folders

Test: Class 0: 7426
      Class 1: 631
Train: Class 0: 928
       Class 1: 80
Val: Class 0: 928
     Class 1: 79

I have used an 80/10/10 split for train/test/val respectively.

Can someone guide me on what to do so that I can ensure the accuracy is 95%+ and the validation loss graph is less erratic?

  1. I am thinking of tuning the learning rate though it doesn't seem to be working by much.
  2. Another suggestion is to use test time augmentation.
  3. Also, the link on fast.ai has a comment like so

That is also part of the reasons why a weighted ensemble of different performing epoch models will usually perform better than the best performing model on your validation dataset. Sometimes choosing the best model or the best ensemble to generalize well isn’t as easy as selecting the lower loss/higher accuracy model. 4. Should I use L2 regularization in addition to the current dropout?

Applying augmentation of any kind to the validation set is a strict no-no and the dataset is generated by my company which I cannot get more of.

  • $\begingroup$ You're asking too many questions here! Please, focus on one specific problem/question. You can ask subsequent questions in other separate posts! $\endgroup$ – nbro Mar 12 '20 at 13:52
  • $\begingroup$ I am panicking mate. $\endgroup$ – Anshuman Kumar Mar 12 '20 at 13:52
  • $\begingroup$ Please help @nbro $\endgroup$ – Anshuman Kumar Mar 13 '20 at 7:30
  • $\begingroup$ I don't know I if I will be useful. Maybe later I will have a closer look at your problem! $\endgroup$ – nbro Mar 13 '20 at 10:22
  • 1
    $\begingroup$ I've quickly looked at your source code. You're using a dropout rate of 0.75, which means you will be training only 0.25 percent of the network at every iteration. Maybe try reducing it to 0.5 and see if you get any improvement. $\endgroup$ – nbro Mar 14 '20 at 2:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.