0
$\begingroup$

I wanted to train a model that recognizes sign language. I have found a dataset for this and was able to create a model that would get 94% accuracy on the test set. I have trained models before and my main goal is not to have the best model (I know 94% could easiy be tuned up). However these models where always for class exercises and thus were never used on 'real' new data.

So I took a new picture of my hand that I know I wanted to be a certain letter (let's assume A).

Since my model was trained on 28x28 images, I needed to re-size my own image because it was larger. After that I fed this image to my model only to get a wrong classification.

https://imgur.com/a/QE6snTa

These are my pictures (upper-left = my own image (expected class A), upper-right = an image of class A (that my model correctly classifies as A), bottom = picture of class Z (the class my image was classified as)).

You can clearly see that my own image looks for more like the image of class A (that I wanted my model to predict), than the model it did predict.

What could be reasons that my model does not work on real-life images? (If code is wanted I can provide it ofcourse but since I don't know where I go wrong, it seemed out of line to copy all the code).

$\endgroup$
1
$\begingroup$

I’m assuming that you used LeNet (our some other model with small number of parameters) since your training image size is 28x28. Note that LeNet doesn’t generalize well to new images. I think it performs fine (>90%) on MNIST but not good on CIFAR10 (>60%) albeit both datasets contain similar size image. (Just trying to remember the performance from PyTorch implementations). It’s more about if the model has capacity to learn the complexity of dataset. CIFAR10 is more complex and harder to model than MNIST.

LeNet is a small image classification model (in terms of capacity) so it cannot nicely learn the correlation between pixels of input images well and therefore doesn’t perform well on unseen images.

In your case it seems like your model has overfit to training examples. It might perform well on test images because both training and test subsets are sampled from same data generating distribution but real-world images it might experience in future might be different (like your own hand). If it doesn’t perform well on unseen images we say it has not generalized well, which looks the case in your situation. In this case you need a validation set to validate that your model generalizes to unseen images. If you have it then you should use it in early stopping regularization technique. You can also add other regularizers to your model (the simplest one is weight decay).

But instead of inventing your network architecture why don’t you use models like ResNet. Just fine-tune the pre-trained ResNet on your own dataset. I’d prefer to fine-tune personally in this situation because data distribution it was trained on (ImageNet) is pretty different from your hand sign dataset. In other case if your dataset contained nature and surrounding images I’d rather freeze the parameters of fixed-feature extractor layers and trained only the last few layers of ResNet (or similar model).

I hope this helps!

| improve this answer | |
$\endgroup$
1
$\begingroup$

This is not an uncommon situation. The data set your model is trained on represents a certain probability distribution. Your test set is most likely a good representation from that distribution so your test results will be good. However when you use real world images they may or may not have a similar distribution. Typically if the training set is large and diverse it is a good representation of the distribution and when the model is used to classify a real world image it will do so correctly. I think I know of the data set you are working with and if I recall it is fairly large. So the problem may be that your model is not complex enough to fully capture the complexity of the data. You can test that fairly simply by using transfer learning with a model that is known to be effective for image classification. I recommend using the MobileNet model. It contains only about 4 million parameters but is about as accurate as larger models containing 10 times as many parameters. So MobileNet is not computationally expensive to train. Documentation can be found here.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Are you sure the image quality in your test set and phone camera image similar? I once trained a CNN model on poor quality images and with very good validation accuracy but when I tested on image from my camera it didn't work at all. I degraded the image quality by resizing the image from my camera to a small size then again back to required size and it worked perfectly.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.