I implemented a neural network in python from scratch (The github repo to it.).

It works reasonably well and I benchmarked different designs using convolutional layers on MNIST and it worked better than expected, but now after trying it on CIFAR-10, I just get mumbo-jumbo.

My network is a 4-layer network, the first three being convolutional layers, followed by ELUs and a max-pooling layer. The last one is a fully-connected one witha softmax on top.
My input is normalized to 0-1 range, I initialize weights to a standard deviation of 1. (Xavier init) and I use ExponentialLinearUnits.

Despite this, my softmax is completely saturated at initialization (at values either 0. or 1., nothing in between) and it really does not change that much.
Also my network learns very slowly, probably due to the saturated softmax, and all layer activations are fairly large:

  1. conv after pool: -1. - 25.
  2. conv after pool: -1. - 668.
  3. conv after pool: -1. - 4117.
  4. fully connected: 8334. - 12028.
  5. softmax: 0. - 1.

Is this normal behavior? Can I change my architecture to prevent this or initialize in another way? Or do I absolutely have to use batch normalization?

  • $\begingroup$ That is a huge repo I don't have the time to go over, but here's the method I normally use to identify problems. Since you're initialisation seems good, try setting everything back to relu. If that doesn't help, create a very simple architecture of say a 3x3x1 image, with minimal convolutions, maybe 1 max pool and very small FC layers, manually do the calculations by hand based on what the network initialised the weights to be, and see how that compares. There's lots of things that could be going wrong, especially since the typical problems (initialisation, normalisation) you've said are good. $\endgroup$ – Recessive Jul 25 '19 at 2:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.