The toy problem:
50 unique numbers are randomly selected from number 0 to 99.
- If number 1 appears in the selection but number 2 doesn't, the selection is labelled as "1".
- If number 2 appears in the selection but number 1 doesn't, the selection is labelled as "2".
- If both number 1 and number 2 appear in the selection, the selection is labelled as "3".
- Lastly, if neither 1 nor 2 appears in the selection, the selection is labelled as "0".
Let's say we want to use a fully connected neural network to classify the selections. The number of input nodes would be 50, which is the number of items in each selection. The number of output nodes will be 4, which will be number of labels/classes. The loss function is cross entry loss which is commonly used in classification problems. The activation function for the hidden layers is ReLU(or leaky ReLU).
The question is how do we decide on the number of hidden layers and the number of nodes in each layer? Also let's say we use stochastic gradient descent(SGD) as optimizer, how do we choose the learning rate? Also the training batch size?
I have tried some arbitrary values for those hyper parameters. In some cases the loss doesn't decrease. In some cases the loss decrease to almost 0 for the training data set, but the prediction accuracy for the validation dataset is 25%, which is as good as random guess.
Is there any guidelines/heuristics on how to choose those hyperparameters?