1
$\begingroup$

I am training a classifier to identify 24 hand signs of American Sign Language. I created a custom dataset by recording videos in different backgrounds for each of the signs and later converted the videos into images. Each sign has 3000 images, that were randomly selected to generate a training dataset with 2400 images/sign and validation dataset with the remaining 600 images/sign.

  • Total number of images in entire dataset: 3000 * 24 = 72000
  • Training dataset: 2400 * 24 = 57600
  • Validation dataset: 600 * 24 = 14400
  • Image dimension (Width x Height): 1280 x 720 pixels

The CNN architecture used for training

model = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),
    MaxPooling2D(pool_size=(2,2)),
    Dropout(0.25),

    Conv2D(32, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2,2)),
    Dropout(0.25),

    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2,2)),
    Dropout(0.25),

    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2,2)),
    Dropout(0.25),

    Flatten(),
    Dense(128, activation='relu'),
    Dropout(0.25),

    Dense(NUM_CLASSES, activation='softmax')
])

Training parameters:

IMG_HEIGHT = 224
IMG_WIDTH = 224
BATCH_SIZE = 32
NUM_CLASSES = 24
train_datagen = ImageDataGenerator(rescale = 1./255,
                                   width_shift_range=0.1,
                                   height_shift_range=0.1,
                                   zoom_range=0.1,
                                   fill_mode='constant')
EPOCHS = 20
STEPS_PER_EPOCH = TRAIN_TOTAL // BATCH_SIZE
VALIDATION_STEPS = VALIDATION_TOTAL // BATCH_SIZE

callbacks_list = [
    tf.keras.callbacks.EarlyStopping(monitor = 'accuracy',
                                     min_delta = 0.005,
                                     patience = 3),
    tf.keras.callbacks.ModelCheckpoint(filepath = 'D:\\Models\\HSRS_ThesisDataset_5Mar_1330.h5',
                                       monitor= 'val_loss',
                                       save_best_only = True)
]

optimizer = 'adam'

The model accuracy and model loss graph is shown in the figure below:

Model Accuracy and Loss

The results obtained at the end of the training are

  • Train acc: 0.8000121
  • Val acc: 0.914441

I read this article explaining why the validation loss is lower than the training loss I want to know:

  1. Is it because of the smaller dataset and random shuffling of the images?
  2. Is there any way to improve the condition without changing the dataset?
  3. Will this have a very detrimental effect on the model performance in real test cases? If not, can I just focus on improving the training accuracy of the overall model?
$\endgroup$
1
  • $\begingroup$ That article you're mentioning tells you that dropout can decrease the training performance and you are using dropout. Have you tried not to use dropout during training and see if something changes? $\endgroup$
    – nbro
    Jun 11, 2020 at 11:37

2 Answers 2

3
$\begingroup$
  1. Assuming you pass through the entire validation dataset, this can't be due to shuffling since you still compute the loss/accuracy over the entire dataset, so order does not really matter here. It is more likely that you have a significantly smaller or less representative validation dataset, e.g., distribution of the validation dataset can be skewed towards classes where your model performs better.
  2. What do you mean exactly by improving the situation? Having a better validation accuracy is not necessarily bad. In any case, if you decrease the effect of regularization, e.g., lowering weight decay, training accuracy might go up but your model might generalize worse, i.e., you might get a lower validation accuracy.
  3. No, the goal of training is never to maximize training accuracy. You can trivially do so by just memorizing the training dataset. In short, the goal of training is to get good generalization and as long as you get a satisfactory validation accuracy, it is likely that this has happened to some degree (assuming you have a good validation dataset of course).
$\endgroup$
4
  • $\begingroup$ Thank you for the reply. What does it mean to have good representative validation dataset? Just to understand it, my images were taken on plain and complex backgrounds, so both these types of images are present in the training and validation dataset. Should I segregate them like - Training data with plain bakgrounds and Validation dataset with more complex bakgrounds? $\endgroup$ Mar 12, 2020 at 9:19
  • $\begingroup$ Although I see higher accuracies, it performs significantly poor on several test images I checked. So, it leads to a confusion to how much should I train? $\endgroup$ Mar 12, 2020 at 12:09
  • 1
    $\begingroup$ Briefly, the format of the images and the distribution of the classes in your validation set should ideally be the same as your training dataset. You can use the 'early stopping' criteria to decide when to stop training. That is, just keep training as long as your validation loss decreases. Then, use the model snapshotted at lowest validation loss to do inference on test data. Hope that helps. $\endgroup$
    – SpiderRico
    Mar 13, 2020 at 2:29
  • $\begingroup$ Regarding the equal distribution of data for each class, I do not think it is a problem since I have segregated them in different folders. So each class has 2400 training images, 600 validation images. What do you mean by format? The resolutions of all the images are the same. Thanks for the tip to use 'val_loss' as a metric in earlystopping. $\endgroup$ Mar 13, 2020 at 7:46
0
$\begingroup$
Validation dataset: 600 * 24 = 14400

Means that you are augmenting the validation set, right? For an experiment, you can do that and it might take validation accuracy more than train accuracy?

The idea of augmentation in only valid for the training set and you should not change the validation set or test set.

You can try without the augmentation in the validation set and see the result.

$\endgroup$
1
  • $\begingroup$ No, I have not augmented the validation dataset. I have 600 (images/hand sign) x 24 (24 hand signs) = 14400 validation dataset images $\endgroup$ Mar 12, 2020 at 9:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .