The toy problem:
50 unique numbers are randomly selected from number 0 to 99.

  • If number 1 appears in the selection but number 2 doesn't, the selection is labelled as "1".
  • If number 2 appears in the selection but number 1 doesn't, the selection is labelled as "2".
  • If both number 1 and number 2 appear in the selection, the selection is labelled as "3".
  • Lastly, if neither 1 nor 2 appears in the selection, the selection is labelled as "0".

    Let's say we want to use a fully connected neural network to classify the selections. The number of input nodes would be 50, which is the number of items in each selection. The number of output nodes will be 4, which will be number of labels/classes. The loss function is cross entry loss which is commonly used in classification problems. The activation function for the hidden layers is ReLU(or leaky ReLU).

    The question is how do we decide on the number of hidden layers and the number of nodes in each layer? Also let's say we use stochastic gradient descent(SGD) as optimizer, how do we choose the learning rate? Also the training batch size?

    I have tried some arbitrary values for those hyper parameters. In some cases the loss doesn't decrease. In some cases the loss decrease to almost 0 for the training data set, but the prediction accuracy for the validation dataset is 25%, which is as good as random guess.

    Is there any guidelines/heuristics on how to choose those hyperparameters?

1 Answer 1


Although you label it as a toy system, I see three possible ways to simplify it and get the classifier to start working.

  1. Instead of starting with 50 inputs, start with only 2 inputs.
  2. Instead of considering a range from 0 to 99, start with a smaller range. Since we are to learn a threshold between 2 and 3, a symmetric range would suggest having the numbers in the range 0..5, so the data are not that biased towards the trimmed region.
  3. Instead of viewing this as a classifier, take it as a function approximator: you want the sum of the two inputs when they are different and the maximum when they are the same.

Once we get this small cell to work, we can take at least the following two possible approaches. The basic cell probably needs to have some hidden layer(s) already.

Fully connected. Double the number of inputs while the system works. When it crashes, add an intermediate layer. How many nodes? I guess the ceiling of the square root of the number of input cells, not sure why.

Sparse. Place a basic cell at the output and double the number of cells left-wards, until you can take all your input. For 50 inputs, layers with 32, 16, 8, 4, 2, 1 cells should do the job. Note that as each cell takes two inputs, 32 input cells provide 64 inputs, larger than your required 50. Although this network will be deeper than the fully connected alternative, the number of training parameters is given by the basic cell, so it is actually smaller. I take the idea of doubling the number of cells as a tree from this video by Andrew Ng.

As for the learning rate for the SGD algorithm, you may want to use the default value 0.01. For the batch size, you want to learn a function of two variables with nine possible values after trimming, (0,1,2)x(0,1,2). I think I would start with say 16 data points and double while needed, or start at 10 and multiply by 10. But it still surprises me how much data is needed to learn a simple function, so my starting points are likely too low.

Do you have a Colab notebook with your attempt?

  • 1
    $\begingroup$ Thanks for your answer. I am still trying to digest your answer and suggestions. But here is the link to colab that I have created for this. colab.research.google.com/drive/… $\endgroup$
    – Yang
    Commented Mar 24, 2023 at 10:55
  • $\begingroup$ Ok, thanks! I am more familiar with TensorFlow, but this is a nice opportunity to see some PyTorch. I'll have a look and try some changes asap. $\endgroup$ Commented Mar 24, 2023 at 11:13
  • $\begingroup$ Thanks. No hurry. I am also just starting to learn these libraries, and this tutorial made me decided to go with pytorch. pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html About the suggestion to start with simpler cases, I was thinking if the cases are too simple, say only 2 inputs, there are only a very limited number of input combinations. In this case does it mean that the NN can simply exhaust all the possibilities and "remember" the outputs? $\endgroup$
    – Yang
    Commented Mar 24, 2023 at 12:09
  • $\begingroup$ Well... In a sense, it has to remember the outputs, doesn't it? Should it forget the outputs and give wrong results? Maybe I didn't get the aim of your project, but it brings to mind modeling logical functions such as OR, AND and XOR with networks as small as possible. $\endgroup$ Commented Mar 24, 2023 at 12:29
  • $\begingroup$ By "remember" I mean it has seen all the input combinations in the training dataset. In this case the NN just behave like a lookup table? What we want to do is to use it to predict the output for some inputs that it has not seen before, but hopefully it has extracted certain useful patterns for any inputs in general. $\endgroup$
    – Yang
    Commented Mar 24, 2023 at 12:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .