I wanted to understand back-propagation so I made a basic neural network library. I used momentum, with learning rate = $0.1$, beta = $0.99$, epochs = $200$, batch size = $10$, loss function is cross entropy and model structure is $784$, $64$, $64$, $10$ and all layers use sigmoid. It performed terribly at first, so I initialized all the weights and biases in the range $[10^{-9}, 10^{-8}]$ and it worked. I am quite new to deep learning and I find TensorFlow doesn't seem as friendly to beginners who want to play around with hyper-parameters. How do you find the right hyper-parameters? I trained it on 100 digits (which took 10 minutes), tweaked hyper-parameters, chose the best set and trained the model using that set on the entire data set of $60,000$ images. I also found that halving the epochs and doubling the training set size gave better results. Are there fool proof heuristics to find good hyper-parameters? What is the best set of hyper-parameters (without regularization, dropout, etc) for MNIST digits? Here is the code for those who want to take a look.

  • $\begingroup$ Also, how is TensorFlow super fast? How does training in TensorFlow result in higher accuracy while training a custom network using same parameters take longer? I am not utilizing my GPU or anything so what gives? $\endgroup$ – Karthik Jul 3 '20 at 7:59

There is no singular best set of hyperparameters. Even more, there is no real search algorithm for hyperparameters. You can do a grid search, but this obviously will take some time. Most people either do that or will try to handpick their parameters.

A few other things to note: Initializing your weights at [10^9,10^8] doesn't seem right to me. They should be centered and close to zero initially. You should also take a look again how to set up train/test/val splits.

  • $\begingroup$ I meant the range $[10^{-9}, 10^{-8}]$. It was a typo. But almost all MNIST tutorials using TensorFlow seem to use the 6:1 train-test split ratio so I went for it. $\endgroup$ – Karthik Jul 3 '20 at 9:40
  • $\begingroup$ i was confused by the word set chose the best set and trained the model using that set on the entire data set but you are right, it is a good ratio. At the end of the day, good hyperparameters are most of the time default ones with actions taken to minimize overfitting. $\endgroup$ – N. Kiefer Jul 3 '20 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.