# 96.91% accuracy on MNIST after 2 hours of training using custom made neural net library. Ways to improve?

I wanted to understand back-propagation so I made a basic neural network library. I used momentum, with learning rate = $$0.1$$, beta = $$0.99$$, epochs = $$200$$, batch size = $$10$$, loss function is cross entropy and model structure is $$784$$, $$64$$, $$64$$, $$10$$ and all layers use sigmoid. It performed terribly at first, so I initialized all the weights and biases in the range $$[10^{-9}, 10^{-8}]$$ and it worked. I am quite new to deep learning and I find TensorFlow doesn't seem as friendly to beginners who want to play around with hyper-parameters. How do you find the right hyper-parameters? I trained it on 100 digits (which took 10 minutes), tweaked hyper-parameters, chose the best set and trained the model using that set on the entire data set of $$60,000$$ images. I also found that halving the epochs and doubling the training set size gave better results. Are there fool proof heuristics to find good hyper-parameters? What is the best set of hyper-parameters (without regularization, dropout, etc) for MNIST digits? Here is the code for those who want to take a look.

• Also, how is TensorFlow super fast? How does training in TensorFlow result in higher accuracy while training a custom network using same parameters take longer? I am not utilizing my GPU or anything so what gives? Jul 3, 2020 at 7:59

A few other things to note: Initializing your weights at [10^9,10^8] doesn't seem right to me. They should be centered and close to zero initially. You should also take a look again how to set up train/test/val splits.
• I meant the range $[10^{-9}, 10^{-8}]$. It was a typo. But almost all MNIST tutorials using TensorFlow seem to use the 6:1 train-test split ratio so I went for it. Jul 3, 2020 at 9:40
• i was confused by the word set chose the best set and trained the model using that set on the entire data set but you are right, it is a good ratio. At the end of the day, good hyperparameters are most of the time default ones with actions taken to minimize overfitting. Jul 3, 2020 at 9:47