One vs multiple output neurons

Question

Consider an MLP that outputs an integer 'rating' of 0 to 4. Would it be correct to say this could be modeled in either of the following ways:

1. map each rating in the dataset to a 'normalized set' between 0 and 1 (i.e. 0, 0.25, 0.5, 0.75, 1), have a single neuron with sigmoid activation at output provide a single decimal value and then take as the rating whatever is closest to that value in the 'normalized set'

2. have 5 output neurons with a softmax activation function output 5 values, each representing a probability of one of the 5 ratings as the outcome, and then take as the rating whichever neuron gives the highest probability?

If this is indeed the case, how does one typically decide 'which way to go'? Approach 1 certainly appears to yield a simpler model. What are the considerations, pros/cons of each approach? Perhaps a couple of concrete examples to illustrate?

Answer 1

This depends on whether the output is a continuous or discrete variable. If the output variable is discrete (there are a finite number of possibilities that it can be), as in a classification task (such as this one, where you are trying to place the input into one of 5 categories), you want to use one output neuron for each class. If the variable is continuous, however, you should only use one output neuron.

This is because of how the training process works. When training your network successively makes adjustments to try and reduce the errors. These adjustments are made in the direction of the error – so if the network predicts a value which is too high then the network's weights are adjusted to make the output value lower. On the other hand if the network's predicts a value which is too low the network's weights are adjusted to make the output bigger.

If you have output neurons labeled 0 to 4 and a training sample with some input value and a target prediction of 2 then the neural network will make its prediction. Once the prediction has made each neuron is adjusted individually – in this case neuron 2 will be adjusted in the direction of the correct probability and all the other neurons will be adjusted in the direction of the incorrect probability. In this way you have one prediction for each class.

Backpropagation is a about error attribution, and using multiple neurons allows the error of the neural network to be better attributed as the neural network can adjust each neuron individually, and thus adjust the required probabilities for each class.

Using a single neuron with a sigmoid activation function would be less good as the sigmoid function saturates values close to 0 and 1 so there would be an unnatural bias towards category 0 and category 4 over the other categories. The neural network could learn to overcome this, but it would take more time.

Answer 2

In the case of one output neuron, you don't have to use sigmoid. As Teymour Aldridge suggested, it would cause a tendency to output 0 or 1. What I normally do is that I set the layer before the output layer to sigmoid of tanh so it won't output ridiculously off numbers, and set the output layer to linear. There would be cases that it outputs something like 1.5, but over time that disappears.

Hope it helps :)

Answer 3

A somewhat large set of designs and set-ups can be made to learn a rating function for a given set of labeled examples. If the objectives are simplicity and effectiveness (accuracy, reliability, and speed), then a third option should be considered.

The requirement in the question includes, "Outputs an integer rating 0 [through] 4 [inclusive]." For such a discrete result, the number of required output bits $b$ (where $s$ is the number of possible states and $I$ is the set of integers) is given as follows.

$$\min_b \, (b \in I \; \land \; b \ge \log_2 s)$$

In this case, we require three bits of output.

$$s = 5 \quad \implies \quad b = 3$$

Note that with similar configuration ratings of 0 through 7 would also require only three bits of output. Either way, the output layer would likely be simplest and most efficient if its activation function was binary step function. This removes the need for rounding after it is applied. The output layer would then provide a binary value indicating rating. The goal of learning would be to reduce the error between the feed forward output and the associated the binary value of the label for each example.

Previous layer(s) could be sigmoid or a more contemporary and less problematic continuous activation function like ISRLU.

Since the engineer can select the error function used by the learning framework to accept any input range and distribution, normalizing the labels for supervised learning is primarily employed to remove redundancy from time and resource consuming operations required to compute error. With ratings as the labels, unless the distribution of ratings is skewed and the data set is such that learning time is excessive, normalization may not be necessary. If it is, it would likely be because improving the label distribution in advance (requiring floating point input to the error function and removing skew) would reduce learning time.

The other two approaches introduce unnecessary complexities mentioned in context above. A consequence of removing complexities without adding impediments to convergence is more efficiency during learning and during execution after learning.