I implemented a neural network in python from scratch (The github repo to it.).
It works reasonably well and I benchmarked different designs using convolutional layers on MNIST and it worked better than expected, but now after trying it on CIFAR-10, I just get mumbo-jumbo.
My network is a 4-layer network, the first three being convolutional layers, followed by ELUs and a max-pooling layer. The last one is a fully-connected one witha softmax on top.
My input is normalized to 0-1 range, I initialize weights to a standard deviation of 1. (Xavier init) and I use ExponentialLinearUnits.
Despite this, my softmax is completely saturated at initialization (at values either 0. or 1., nothing in between) and it really does not change that much.
Also my network learns very slowly, probably due to the saturated softmax, and all layer activations are fairly large:
- conv after pool: -1. - 25.
- conv after pool: -1. - 668.
- conv after pool: -1. - 4117.
- fully connected: 8334. - 12028.
- softmax: 0. - 1.
Is this normal behavior? Can I change my architecture to prevent this or initialize in another way? Or do I absolutely have to use batch normalization?