2
$\begingroup$

I developed a CNN for image analysis. I've around 100K labeled images. I'm getting a accuracy around 85% and a validation accuracy around 82%, so it looks like the model generalize better than fitting. So, I'm playing with different hyper-parameters: number of filters, number of layers, number of neurons in the dense layers, etc.

For every test, I'm using all the training data, and it is very slow and time consuming.

Is there a way to have an early idea about if a model will perform better than another?

$\endgroup$
4
  • $\begingroup$ Hi, you don't specify the expected accuracy in your question. The difference that you have written between 85% for training and 82% for validation seems normal and implies that the model generalises well. What is your expected accuracy, and how do you know it is expected (is there another experiment using the exact same data that you are trying to match)? $\endgroup$ Sep 6 '18 at 13:15
  • $\begingroup$ @NeilSlater yes, I think that it generalizes ok, even if I try to overfit I don't get better acc that 87% or so. Trained human accuracy could be around 98%. I've tried to use transfer learning with similar or worst results... The labeled data is made by my, no another experiment is using the exact data. $\endgroup$
    – Mquinteiro
    Sep 6 '18 at 13:21
  • $\begingroup$ So your expectation of 98% is based on human accuracy on the same task? $\endgroup$ Sep 7 '18 at 18:52
  • $\begingroup$ @NeilSlater yes it is, or something close to it, 95% could be enough, but I'm far, now playing manually with the number of convolution layers and filters. so tedious $\endgroup$
    – Mquinteiro
    Sep 7 '18 at 22:42
1
$\begingroup$

The simple answer to your question is "No" with a caveat.

The caveat is that there are signs that your network is never going to perform well. For example, the epoch accuracy fails to improve or even consistently declines over the first several epochs, or the validation accuracy is flat or declining. It could be that the validation loss starts high and just keeps increasing from the beginning. These are all bad signs.

Outside of this, however, it's very tough to know the model won't work well in the long run. For example, we have a model we built for solving a set of CAPTCHAs. The regression portions of that converged very quickly, but the portions that solved the rest of the CAPTCHA took something like 18 hours before they converged. Honestly, we only ran it that long because it was the end of the day and the regression piece looked so promising; there was nothing in the training behavior of the CAPTCHA solver that looked like it would work (even though our intuition was that it should.)

In the end, we have a 96%+ accuracy CAPTCHA solver that we likely would have killed if we had watched it train for more than 10 or 15 minutes.

$\endgroup$
0
$\begingroup$

I would like you to try with following changes in the model.

  • Introduce batch normalization layer in the model.
  • Try with the batch size of 32 - 64
  • Use different architecture, Like VGG, Resnet, etc..

There is no full proof answer to this question but you can get best by trying some know strategies.

$\endgroup$
1
  • $\begingroup$ Actually I did all your say 1) BN is the network. 2) The batch size is 50, tested with VGG19 and the results are similar to mine, also I have test with Vision AutoML from google with similar results. $\endgroup$
    – Mquinteiro
    Sep 6 '18 at 14:47
0
$\begingroup$

Your description sounds like something similar to Imagenet dataset. According to this website, the state-of-the-art top-1 accuracy just as high as 86%, not much higher than yours. There are plenty of methods to improve accuracy. I would suggest you to read the paper or github listed in SOTA to find ideas that best fits your situation.

$\endgroup$
0
$\begingroup$

I can't comment but here are a few suggestions:

  • Play with lr and lr finder
  • Use a pretrained model
  • Use architecture search or use something like efficientnet b6
  • Use swish over relu
  • Try different optimizers
  • Try Bayesian Optimization
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.