# How do I deal with an erratic validation set loss when the loss on my training set is monotonically decreasing?

Model used

mobilenet_model = MobileNet(input_shape=in_dim, include_top=False, pooling='avg', weights='imagenet')
mob_x = Dropout(0.75)(mobilenet_model.output)
mob_x = Dense(2, activation='sigmoid')(mob_x)

model = Model(mobilenet_model.input, mob_x)

for layer in model.layers[:50]:
layer.trainable=False

for layer in model.layers[50:]:
layer.trainable=True

model.summary()


The rest of the code

in_dim = (224,224,3)
batch_size = 64
samples_per_epoch = 1000
validation_steps = 300
nb_filters1 = 32
nb_filters2 = 64
conv1_size = 3
conv2_size = 2
pool_size = 2
epochs = 20
classes_num = 2
lr = 0.000004
train_datagen = ImageDataGenerator(
rescale=1. / 255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
'output/train',  # this is the target directory
target_size= in_dim[0:2],  # all images will be resized to 224*224
batch_size=batch_size,
class_mode='categorical')
#Found 6062 images belonging to 2 classes.
validation_generator = test_datagen.flow_from_directory(
'output/val',
target_size=in_dim[0:2],
batch_size=batch_size,
class_mode='categorical')
#Found 769 images belonging to 2 classes.
from keras.callbacks import EarlyStopping
#set early stopping monitor so the model stops training when it won't improve anymore
early_stopping_monitor = EarlyStopping(patience=3)
steps_per_epoch = 10
from keras import backend as K

def recall_m(y_true, y_pred):
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
recall = true_positives / (possible_positives + K.epsilon())
return recall

def precision_m(y_true, y_pred):
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
precision = true_positives / (predicted_positives + K.epsilon())
return precision

def f1_m(y_true, y_pred):
precision = precision_m(y_true, y_pred)
recall = recall_m(y_true, y_pred)
return 2*((precision*recall)/(precision+recall+K.epsilon()))

history = model.fit_generator(
train_generator,
steps_per_epoch=2000// batch_size ,
epochs=50,
validation_data=validation_generator,
validation_steps=800// batch_size,
callbacks = [early_stopping_monitor],
)
test_generator = train_datagen.flow_from_directory(
'output/test',
target_size=in_dim[0:2],
batch_size=batch_size,
class_mode='categorical')
loss, accuracy, f1_score, precision, recall = model.evaluate(test_generator)
print("The test set accuracy is ", accuracy)
#The test set accuracy is  0.9001349538122272


From what I have gathered from this post and this article, I understand that the validation set is much smaller with respect to the training set. I have applied augmentation to the test set due to this and that boosted test set accuracy by 1%.

Please note that the test train split is "stratified" as here is a breakdown of each individual class in test/train/validation folders

Test: Class 0: 7426
Class 1: 631
Train: Class 0: 928
Class 1: 80
Val: Class 0: 928
Class 1: 79


I have used an 80/10/10 split for train/test/val respectively.

Can someone guide me on what to do so that I can ensure the accuracy is 95%+ and the validation loss graph is less erratic?

1. I am thinking of tuning the learning rate though it doesn't seem to be working by much.
2. Another suggestion is to use test time augmentation.
3. Also, the link on fast.ai has a comment like so

That is also part of the reasons why a weighted ensemble of different performing epoch models will usually perform better than the best performing model on your validation dataset. Sometimes choosing the best model or the best ensemble to generalize well isn’t as easy as selecting the lower loss/higher accuracy model. 4. Should I use L2 regularization in addition to the current dropout?

Applying augmentation of any kind to the validation set is a strict no-no and the dataset is generated by my company which I cannot get more of.

• You're asking too many questions here! Please, focus on one specific problem/question. You can ask subsequent questions in other separate posts! – nbro Mar 12 at 13:52
• I am panicking mate. – Anshuman Kumar Mar 12 at 13:52