I'm making a custom neural network framework (in C++, if that is of any help). When I train the model on MNIST, depending on how happy the network is feeling, it'll give me either 90%+ accuracy, or get stuck at 10-9% (on validation set).
I shuffle all my data before feeding it to the neural net.
Is there a better randomizer I should be using, or maybe I am not initializing my weights properly (Using
srand to generate values between +/-0.1). Did I somehow hit a saddle point?
My network consists of 784 size input layer, 256, 64, 32, 16 neuron hidden layers, all with RELU, and 10 output with SMAX
Where should I start investigating based on this kind of behavior, when I can't even replicate what is going on?