Can I use 4 neurons for output layer to classify hand written digit?

Question

Hello world of ANN usually uses MNIST hand written digit data. For classes there are 10, therefore it takes 10 neurons in the output layer, each class is 0 to 9 handwritten digit images.

If in the end there is only one active neuron in the output layer, why does not use only 4 neurons in the output layer where each neuron represents a binary digit, so that 16 classes will be more than enough (10 classes).

For example, if the neuron values ​​after the activation function in the output layer are successive like this

 0.1 0.2 0.8 0.9

Then it can be rounded to:

 0 0 1 1

Or instead of being rounded up manually, Why does not use the binary step activation function on the hidden layer before the output layer.

So the prediction result is 3 because 0011 if converted to decimal is 3.

By using 4 neurons, less computational load should be used than using 10 neurons for each class.

So can I use only 4 neurons only in output layer to classify 10 handwritten digit (10 classes).

Below picture is just sweet picture that represent 10 neurons for every class in output layer:

Answer 1

One problem that I see is that you can no longer use the cross-entropy loss function for training, or at least I am not sure how you could do it. This cost function has many advantages, one of them being that you can interpret the activation of the output neurons as the probability of the input being that category (they all sum to one). You can read about some other advantages here: https://kobejean.github.io/machine-learning/2019/07/22/cross-entropy-loss/

Answer 2

Of course you can but I'd not recommend doing this way.

First - it is not part of ML because it is straight logic and should not be learned, so I don't think backpropagation or other algorithms continue performing correctly that way. That way I believe your model could be fitted to avoid middle numbers and respond with only lowest or biggest. And that is the question to you if you really need that.

Second - I suppose it is better to keep the model not for super specific task but for common tasks and be adaptive for cases if you want to gain more details (statistics for example).

The model with output of 10 neurons will allow you to know how sure it is in its answer - that way you may want to behave the other way or not do action at all.

The other reason is in clearness - better to keep parts of the system splitted, that way in case of problems you'll find the root of the problem faster.

Answer 3

I believe it is because when using the binary output, it makes the neurons dependent on each other, but technically it should not be, because it complicates two problems:

• The first problem is how would you calculate the loss function for that? In your case 0011 translates to 3, which means that the third one and fourth one depend on each other, so it is "both neuron 3 and 4 must be 1 at the same time". As far as I am aware, there is no existing loss function for output-dependent binary outputs. The reason lies in the second problem below.
• The second problem is that it simply complicates thing that can be done in a multi-class manner. Even in your case, the simplest implementation is to have 16 neurons, each corresponds to a combination of four binary numbers. For example, if neuron 0 is activated, it is 0000, and if it is neuron 1, it is 0001, and so on. This implementation can use the standard cross-entropy loss, which we know it currently works very well.

The ImageNet dataset has 1000 classes, but the state-of-the-art is already 91% over 100k validation images. Since the 1000 neuron outputs work very well, I don't think they even bother experimenting with 10 neurons (for $2^{10}$ case).