I am developing a CNN model to recognize 24 hand-signs of American Sign Language. I have 2500 Images/hand-sign. The data split is:
Training = 1250 Images/hand-sign
Validation = 625 Images/hand-sign
Testing = 625 Images/hand-sign

How should I proceed with training the model?:
1. Should I develop a model starting from fewer hand-signs (like 5) and then increase them gradually?
2. Should I start models from scratch or use transfer learning (VGG16 or other)
Applying data augmentation, I did some tests with VGG16 and added a dense classifier at the end and received these accuracies:
Train: 0.87610877
Validation: 0.8867307
Test: 0.96533334

Accuracy and Loss Graph

Test parameters:
Framework = Keras, Tensorflow


model = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),

    Conv2D(64, (3, 3), activation='relu'),

    Conv2D(128, (3, 3), activation='relu'),

    Conv2D(256, (3, 3), activation='relu'),

    Conv2D(512, (3, 3), activation='relu'),

    Dense(512, activation='relu'),

    Dense(NUM_CLASSES, activation='softmax')

If I try images with slightly different background and predict the classes (predict_classes()), I do not get accurate results. Any suggestions on how to make the model robust?

  • $\begingroup$ Please, next time, focus on a single problem and questions. For each of your questions, open a different thread/post. $\endgroup$
    – nbro
    Commented Mar 13, 2020 at 14:22

1 Answer 1


I feel your problem might not be with the model itself but with the dataset. If you only have $2500$ images for $24$ labels (hand-signs) that gets you roughly $104$ images per label. This is very little for the models I train (~$80K$ images in the smallest of cases). In my view you got a really decent accuracy at validation and test time for the size of your dataset.

But answering your questions:

  • Starting from few labels and extending is usually helpful when your model is too deep and suffers from convergence problems. Your model is simple enough not to suffer those problems so I would go learning the $24$ labels at once.
  • Transfer learning can help a lot at reducing training times. For example, if you start from a VGG classifier that detects hands there is a good chance that the weights of your convolutional layers are already almost configured for your use case.

Generally speaking, the easiest way to increase the accuracy of your model is to use one of these 2 methods:

  • Increase the dataset: I am not sure if it is possible in your case, maybe you can use image augmentation (rotation, zooming in/out, changes in the color space...).
  • Increase the depth of your model (provided that you have a big dataset).

If you already fulfilled those items, then you can go and make changes to the model architecture or loss function. Looking at your model and from the top of my head I would try to add Batch-Normalization for the convolutional layers.

Hope this helps a bit :)

  • $\begingroup$ Thanks. I am actually having 2500 images for each hand-sign (each label). So total images are around 60000. I have used data augmentation before training the above model. I will try VGG16 classifier like: model = Sequential([ vgg16_base, Dropout(0.2), Flatten(), Dropout(0.2), Dense(512, activation='relu', kernel_regularizer=l2(0.01)), Dense(NUM_CLASSES, activation='softmax') ]) Also, I would unfreeze the last one/two layers. But should I add any other Conv2D, Maxpool, Flatten, or Dense Layers at the end? $\endgroup$ Commented Feb 13, 2020 at 9:53
  • $\begingroup$ Ah really nice then! 60K images is ok. I feel you could train VGG starting from some pretrained weights (for ILSVRC for example) instead of freezing layers. Regarding adding layers, you should add Conv2D in the Conv2D stage before flattening to the Dense stage (same with Dense, add them only in the Dense layers stage). And yes, you could make your model deeper since you have enough training data that normally boost performance because it allows to learn more complex nuances of the use case. $\endgroup$
    – JVGD
    Commented Feb 13, 2020 at 11:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .