1
$\begingroup$

Are there any methods of regularisation of deep neural networks, particularly CNNs (or generally ANN but that will also work on CNNs) that are related only to the network's architecture and not the training itself?

I mean maybe something like how deep they are, amount of conv/pooling/fully connected layers, size of filters, size of steps of filters, etc. any pointers that would help with regularisation.

EDIT: To explain deeper what I mean I might add that I am exploring an experimental idea for the training of the CNNs that is not in any way related to typical gradient descent with backpropagation. That is why typical methods related to training will not work. I can see already that the models train satisfactorily on the training set but don't perform that well on a test set and since I didn't figure out any regularization methods for this type of training I thought maybe there are some related to architecture, that the training process will have to abide.

$\endgroup$
4
  • $\begingroup$ I am not sure I understand your question. If you use e.g. L2 regularization, you're limiting the values that the weights can take, so, in this way, this type of regularization is related to the "architecture" of the neural network. So, can you clarify what you're really asking and why regularization techniques like "L2" or "dropout" are not the answers to your question? $\endgroup$ – nbro Mar 16 at 10:55
  • $\begingroup$ Without more info about your training procedure, it's still not clear (at least to me), why dropout does not work in your case. Have you already tried it? $\endgroup$ – nbro Mar 17 at 13:52
  • $\begingroup$ @nbro The training is based on evolutionary algorithms with a very specific domain based fitness function without use of standard error metrics such as binary crossentropy or MSE. I can't even imagine where would dropout or L2 go into this process. $\endgroup$ – Makintosz Mar 18 at 8:25
  • $\begingroup$ maxnorm over weight vectors is a regularisation technique that is not architectural, but may help in your situation $\endgroup$ – Neil Slater Apr 26 at 20:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.