How to approach a toy classification problem using a neural network?

Question

The toy problem:
50 unique numbers are randomly selected from number 0 to 99.

• If number 1 appears in the selection but number 2 doesn't, the selection is labelled as "1".
  
• If number 2 appears in the selection but number 1 doesn't, the selection is labelled as "2".
  
• If both number 1 and number 2 appear in the selection, the selection is labelled as "3".
  
• Lastly, if neither 1 nor 2 appears in the selection, the selection is labelled as "0".
  
  Let's say we want to use a fully connected neural network to classify the selections. The number of input nodes would be 50, which is the number of items in each selection. The number of output nodes will be 4, which will be number of labels/classes. The loss function is cross entry loss which is commonly used in classification problems. The activation function for the hidden layers is ReLU(or leaky ReLU).
  
  The question is how do we decide on the number of hidden layers and the number of nodes in each layer? Also let's say we use stochastic gradient descent(SGD) as optimizer, how do we choose the learning rate? Also the training batch size?
  
  I have tried some arbitrary values for those hyper parameters. In some cases the loss doesn't decrease. In some cases the loss decrease to almost 0 for the training data set, but the prediction accuracy for the validation dataset is 25%, which is as good as random guess.
  
  Is there any guidelines/heuristics on how to choose those hyperparameters?

Answer 1

Although you label it as a toy system, I see three possible ways to simplify it and get the classifier to start working.

1. Instead of starting with 50 inputs, start with only 2 inputs.
2. Instead of considering a range from 0 to 99, start with a smaller range. Since we are to learn a threshold between 2 and 3, a symmetric range would suggest having the numbers in the range 0..5, so the data are not that biased towards the trimmed region.
3. Instead of viewing this as a classifier, take it as a function approximator: you want the sum of the two inputs when they are different and the maximum when they are the same.

Once we get this small cell to work, we can take at least the following two possible approaches. The basic cell probably needs to have some hidden layer(s) already.

Fully connected. Double the number of inputs while the system works. When it crashes, add an intermediate layer. How many nodes? I guess the ceiling of the square root of the number of input cells, not sure why.

Sparse. Place a basic cell at the output and double the number of cells left-wards, until you can take all your input. For 50 inputs, layers with 32, 16, 8, 4, 2, 1 cells should do the job. Note that as each cell takes two inputs, 32 input cells provide 64 inputs, larger than your required 50. Although this network will be deeper than the fully connected alternative, the number of training parameters is given by the basic cell, so it is actually smaller. I take the idea of doubling the number of cells as a tree from this video by Andrew Ng.

As for the learning rate for the SGD algorithm, you may want to use the default value 0.01. For the batch size, you want to learn a function of two variables with nine possible values after trimming, (0,1,2)x(0,1,2). I think I would start with say 16 data points and double while needed, or start at 10 and multiply by 10. But it still surprises me how much data is needed to learn a simple function, so my starting points are likely too low.

Do you have a Colab notebook with your attempt?