2
$\begingroup$

Consider an MLP that outputs an integer 'rating' of 0 to 4. Would it be correct to say this could be modeled in either of the following ways:

  1. map each rating in the dataset to a 'normalized set' between 0 and 1 (i.e. 0, 0.25, 0.5, 0.75, 1), have a single neuron with sigmoid activation at output provide a single decimal value and then take as the rating whatever is closest to that value in the 'normalized set'

  2. have 5 output neurons with a softmax activation function output 5 values, each representing a probability of one of the 5 ratings as the outcome, and then take as the rating whichever neuron gives the highest probability?

If this is indeed the case, how does one typically decide 'which way to go'? Approach 1 certainly appears to yield a simpler model. What are the considerations, pros/cons of each approach? Perhaps a couple of concrete examples to illustrate?

$\endgroup$
1
$\begingroup$

This depends on whether the output is a continuous or discrete variable. If the output variable is discrete (there are a finite number of possibilities that it can be), as in a classification task (such as this one, where you are trying to place the input into one of 5 categories), you want to use one output neuron for each class. If the variable is continuous, however, you should only use one output neuron.

This is because of how the training process works. When training your network successively makes adjustments to try and reduce the errors. These adjustments are made in the direction of the error – so if the network predicts a value which is too high then the network's weights are adjusted to make the output value lower. On the other hand if the network's predicts a value which is too low the network's weights are adjusted to make the output bigger.

If you have output neurons labeled 0 to 4 and a training sample with some input value and a target prediction of 2 then the neural network will make its prediction. Once the prediction has made each neuron is adjusted individually – in this case neuron 2 will be adjusted in the direction of the correct probability and all the other neurons will be adjusted in the direction of the incorrect probability. In this way you have one prediction for each class.

Backpropagation is a about error attribution, and using multiple neurons allows the error of the neural network to be better attributed as the neural network can adjust each neuron individually, and thus adjust the required probabilities for each class.

Using a single neuron with a sigmoid activation function would be less good as the sigmoid function saturates values close to 0 and 1 so there would be an unnatural bias towards category 0 and category 4 over the other categories. The neural network could learn to overcome this, but it would take more time.

$\endgroup$
  • $\begingroup$ Thanks. Various sources such as this suggest that in the case of BINARY classification it is OK to use a single neuron with sigmoid and round to TRUE or FALSE based on a >0.5 and <0.5 criteria. But based on your response, I infer that if we have more classes, this approach is not so OK because referring back to our example, the sigmoid 'squashing' can result in an inability to distinguish between say a 0 and 1 rating, or a 3 and 4 rating. Do I infer correctly? $\endgroup$ – samiant Aug 14 at 5:45
  • $\begingroup$ Yes. If you have only two classes it's fine to use a single output neuron to predict the probability of one of the classes and (as probabilities sum to 1) calculate the probability of the other class as 1 - [predicted probability of the first class]. The network can learn to handle the squashing caused by the sigmoid function but it will learn better and more efficiently using multiple output neurons. $\endgroup$ – Teymour Aldridge Aug 14 at 6:39
0
$\begingroup$

In the case of one output neuron, you don't have to use sigmoid. As Teymour Aldridge suggested, it would cause a tendency to output 0 or 1. What I normally do is that I set the layer before the output layer to sigmoid of tanh so it won't output ridiculously off numbers, and set the output layer to linear. There would be cases that it outputs something like 1.5, but over time that disappears.

Hope it helps :)

$\endgroup$
  • $\begingroup$ I have one hidden layer consisting of sigmoid-ed neurons, and one sigmoid-ed output neuron. This structure is really for 'regression' problems, but based on responses so far, would it be correct to say that it can be 'adapted' for my classification problem by removing the sigmoid on the output neuron so that the result becomes unbounded, and then classify per: rating 0 if <0.5. rating 1 if >0.5 and <1.5. rating 2 if >1.5 and <2.5. rating 3 if >2.5 and <3.5. rating 4 if >3.5. What are the expected disadvantages of doing it this way - eg. less accuracy for the same number of epochs? $\endgroup$ – samiant Aug 15 at 22:58
0
$\begingroup$

A somewhat large set of designs and set-ups can be made to learn a rating function for a given set of labeled examples. If the objectives are simplicity and effectiveness (accuracy, reliability, and speed), then a third option should be considered.

The requirement in the question includes, "Outputs an integer rating 0 [through] 4 [inclusive]." For such a discrete result, the number of required output bits $b$ (where $s$ is the number of possible states and $I$ is the set of integers) is given as follows.

$$\min_b \, (b \in I \; \land \; b \ge \log_2 s)$$

In this case, we require three bits of output.

$$s = 5 \quad \implies \quad b = 3$$

Note that with similar configuration ratings of 0 through 7 would also require only three bits of output. Either way, the output layer would likely be simplest and most efficient if its activation function was binary step function. This removes the need for rounding after it is applied. The output layer would then provide a binary value indicating rating. The goal of learning would be to reduce the error between the feed forward output and the associated the binary value of the label for each example.

Previous layer(s) could be sigmoid or a more contemporary and less problematic continuous activation function like ISRLU.

Since the engineer can select the error function used by the learning framework to accept any input range and distribution, normalizing the labels for supervised learning is primarily employed to remove redundancy from time and resource consuming operations required to compute error. With ratings as the labels, unless the distribution of ratings is skewed and the data set is such that learning time is excessive, normalization may not be necessary. If it is, it would likely be because improving the label distribution in advance (requiring floating point input to the error function and removing skew) would reduce learning time.

The other two approaches introduce unnecessary complexities mentioned in context above. A consequence of removing complexities without adding impediments to convergence is more efficiency during learning and during execution after learning.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.