I was thinking of creating a CNN. Now it is known CNN takes long times to train so it is advisable to stick to known architectures and hyper-parameters.

My question is: I want to tinker with the CNN architecture (since it is a specialised task). One approach would be to create a CNN and check on small data-sets, but then I would have no way of knowing whether the Fully Connected layer at the end is over-fitting the data while the convolutional layers do nothing (since large FC layers can easily over-fit data). Cross Validation is a good way to check it, but it might not be satisfactory (since my opinion is that a CNN can be replaced with a Fully Connected NN if the data-set is small enough and there is little variation in the future data-sets).

So what are some ways to tinker with CNN and get a good estimate for future data-sets in a reasonable training time? Am I wrong in my previous assumptions? A detailed answer would be nice!


1 Answer 1


There is no easy way to play around with the hyper-parameters (number of layers, layer configuration, number of outputs per layer) of a CNN and get an accurate view of how these will affect the resulting performance of your model. However there are a few things that you can do to avoid wasting too much time training and re-training.


When training a CNN we aim to minimize a loss function, thus a better CNN model is defined as one which converges to a set of model parameters with a lower loss function. Identifying the minimum of the loss function for a given CNN is already very difficult, and there is no guarantee that the true minimum will every be reached by fault of gradient descent.

Each variation of the CNN resulting from the different hyper-parameters will still result in a very large number of model parameters. There is no way to know how the loss function will look in this hyper-dimensional space and even harder to estimate its minimum.

What to do?

First, you should try and understand how each layer in the network affects its inputs. You should know what kind of layers to use for what kind of data. You should also know what kinds of activation functions to use for different data distributions. You should also know how many model parameters per layer to try in order to sufficiently compress your data whilst not losing significant information.

You can get a lot of this intuition by reading papers which have found successful models for specific tasks.

In addition, you can train with smaller amounts of data and estimate the potential minimum loss function by seeing how fast the loss function moves towards its minimal value (momentum). Usually a lower minimum is achieved when the loss function decreases faster. However, this is in no way always true. A loss function can converge slowly at first and then speed up later. This is entirely possible. But, you can get some sense of the potential of your model in this way.

  • $\begingroup$ Loss function is actually not a good measure of performance since over fitting can also cause very low losses $\endgroup$
    – user9947
    Commented Aug 30, 2018 at 2:56
  • $\begingroup$ @DuttaA, always make sure to look at the loss function of your validation or test set, not that of your training set. $\endgroup$
    – JahKnows
    Commented Aug 30, 2018 at 23:56

You must log in to answer this question.