1
$\begingroup$

I am training a classifier to identify 24 hand signs of American Sign Language. I created a custom dataset by recording videos in different backgrounds for each of the signs and later converted the videos into images. Each sign has 3000 images, that were randomly selected to generate a training dataset with 2400 images/sign and validation dataset with the remaining 600 images/sign.

  • Total number of images in entire dataset: 3000 * 24 = 72000
  • Training dataset: 2400 * 24 = 57600
  • Validation dataset: 600 * 24 = 14400
  • Image dimension (Width x Height): 1280 x 720 pixels

The CNN architecture used for training

model = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),
    MaxPooling2D(pool_size=(2,2)),
    Dropout(0.25),

    Conv2D(32, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2,2)),
    Dropout(0.25),

    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2,2)),
    Dropout(0.25),

    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2,2)),
    Dropout(0.25),

    Flatten(),
    Dense(128, activation='relu'),
    Dropout(0.25),

    Dense(NUM_CLASSES, activation='softmax')
])

Training parameters:

IMG_HEIGHT = 224
IMG_WIDTH = 224
BATCH_SIZE = 32
NUM_CLASSES = 24
train_datagen = ImageDataGenerator(rescale = 1./255,
                                   width_shift_range=0.1,
                                   height_shift_range=0.1,
                                   zoom_range=0.1,
                                   fill_mode='constant')
EPOCHS = 20
STEPS_PER_EPOCH = TRAIN_TOTAL // BATCH_SIZE
VALIDATION_STEPS = VALIDATION_TOTAL // BATCH_SIZE

callbacks_list = [
    tf.keras.callbacks.EarlyStopping(monitor = 'accuracy',
                                     min_delta = 0.005,
                                     patience = 3),
    tf.keras.callbacks.ModelCheckpoint(filepath = 'D:\\Models\\HSRS_ThesisDataset_5Mar_1330.h5',
                                       monitor= 'val_loss',
                                       save_best_only = True)
]

optimizer = 'adam'

The model accuracy and model loss graph is shown in the figure below:

Model Accuracy and Loss

The results obtained at the end of the training are

  • Train acc: 0.8000121
  • Val acc: 0.914441

I read this article explaining why the validation loss is lower than the training loss I want to know:

  1. Is it because of the smaller dataset and random shuffling of the images?
  2. Is there any way to improve the condition without changing the dataset?
  3. Will this have a very detrimental effect on the model performance in real test cases? If not, can I just focus on improving the training accuracy of the overall model?
$\endgroup$
  • $\begingroup$ That article you're mentioning tells you that dropout can decrease the training performance and you are using dropout. Have you tried not to use dropout during training and see if something changes? $\endgroup$ – nbro Jun 11 at 11:37
2
$\begingroup$
  1. Assuming you pass through the entire validation dataset, this can't be due to shuffling since you still compute the loss/accuracy over the entire dataset, so order does not really matter here. It is more likely that you have a significantly smaller or less representative validation dataset, e.g., distribution of the validation dataset can be skewed towards classes where your model performs better.
  2. What do you mean exactly by improving the situation? Having a better validation accuracy is not necessarily bad. In any case, if you decrease the effect of regularization, e.g., lowering weight decay, training accuracy might go up but your model might generalize worse, i.e., you might get a lower validation accuracy.
  3. No, the goal of training is never to maximize training accuracy. You can trivially do so by just memorizing the training dataset. In short, the goal of training is to get good generalization and as long as you get a satisfactory validation accuracy, it is likely that this has happened to some degree (assuming you have a good validation dataset of course).
| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for the reply. What does it mean to have good representative validation dataset? Just to understand it, my images were taken on plain and complex backgrounds, so both these types of images are present in the training and validation dataset. Should I segregate them like - Training data with plain bakgrounds and Validation dataset with more complex bakgrounds? $\endgroup$ – mayuresh_sa Mar 12 at 9:19
  • $\begingroup$ Although I see higher accuracies, it performs significantly poor on several test images I checked. So, it leads to a confusion to how much should I train? $\endgroup$ – mayuresh_sa Mar 12 at 12:09
  • 1
    $\begingroup$ Briefly, the format of the images and the distribution of the classes in your validation set should ideally be the same as your training dataset. You can use the 'early stopping' criteria to decide when to stop training. That is, just keep training as long as your validation loss decreases. Then, use the model snapshotted at lowest validation loss to do inference on test data. Hope that helps. $\endgroup$ – SpiderRico Mar 13 at 2:29
  • $\begingroup$ Regarding the equal distribution of data for each class, I do not think it is a problem since I have segregated them in different folders. So each class has 2400 training images, 600 validation images. What do you mean by format? The resolutions of all the images are the same. Thanks for the tip to use 'val_loss' as a metric in earlystopping. $\endgroup$ – mayuresh_sa Mar 13 at 7:46
0
$\begingroup$
Validation dataset: 600 * 24 = 14400

Means that you are augmenting the validation set, right? For an experiment, you can do that and it might take validation accuracy more than train accuracy?

The idea of augmentation in only valid for the training set and you should not change the validation set or test set.

You can try without the augmentation in the validation set and see the result.

| improve this answer | |
$\endgroup$
  • $\begingroup$ No, I have not augmented the validation dataset. I have 600 (images/hand sign) x 24 (24 hand signs) = 14400 validation dataset images $\endgroup$ – mayuresh_sa Mar 12 at 9:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.