I was thinking of creating a CNN. Now it is known CNN takes long times to train so it is advisable to stick to known architectures and hyper-parameters.

My question is: I want to tinker with the CNN architecture (since it is a specialised task). One approach would be to create a CNN and check on small data-sets, but then I would have no way of knowing whether the Fully Connected layer at the end is over-fitting the data while the convolutional layers do nothing (since large FC layers can easily over-fit data). Cross Validation is a good way to check it, but it might not be satisfactory (since my opinion is that a CNN can be replaced with a Fully Connected NN if the data-set is small enough and there is little variation in the future data-sets).

So what are some ways to tinker with CNN and get a good estimate for future data-sets in a reasonable training time? Am I wrong in my previous assumptions? A detailed answer would be nice!

  • 1
    $\begingroup$ Side Note: i asked this question on DS.SE but did not get a satisfactory answer. $\endgroup$
    – user9947
    Aug 20 '18 at 14:45
  • $\begingroup$ @quintumnia uuuh what? $\endgroup$
    – user9947
    Aug 21 '18 at 0:55
  • 1
    $\begingroup$ DuttaA ,We have to save our civilisation, by being smart, intelligent and focused.We are here to share our knowledge,which will also be useful to those in the next century.Keep up the spirit because we are not here to make lots of benjamins but rather to be educative and save,the curious soul out there.God Bless. $\endgroup$
    – quintumnia
    Aug 21 '18 at 9:54

There is no easy way to play around with the hyper-parameters (number of layers, layer configuration, number of outputs per layer) of a CNN and get an accurate view of how these will affect the resulting performance of your model. However there are a few things that you can do to avoid wasting too much time training and re-training.


When training a CNN we aim to minimize a loss function, thus a better CNN model is defined as one which converges to a set of model parameters with a lower loss function. Identifying the minimum of the loss function for a given CNN is already very difficult, and there is no guarantee that the true minimum will every be reached by fault of gradient descent.

Each variation of the CNN resulting from the different hyper-parameters will still result in a very large number of model parameters. There is no way to know how the loss function will look in this hyper-dimensional space and even harder to estimate its minimum.

What to do?

First, you should try and understand how each layer in the network affects its inputs. You should know what kind of layers to use for what kind of data. You should also know what kinds of activation functions to use for different data distributions. You should also know how many model parameters per layer to try in order to sufficiently compress your data whilst not losing significant information.

You can get a lot of this intuition by reading papers which have found successful models for specific tasks.

In addition, you can train with smaller amounts of data and estimate the potential minimum loss function by seeing how fast the loss function moves towards its minimal value (momentum). Usually a lower minimum is achieved when the loss function decreases faster. However, this is in no way always true. A loss function can converge slowly at first and then speed up later. This is entirely possible. But, you can get some sense of the potential of your model in this way.

  • $\begingroup$ Loss function is actually not a good measure of performance since over fitting can also cause very low losses $\endgroup$
    – user9947
    Aug 30 '18 at 2:56
  • $\begingroup$ @DuttaA, always make sure to look at the loss function of your validation or test set, not that of your training set. $\endgroup$
    – JahKnows
    Aug 30 '18 at 23:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy