2
$\begingroup$

I am currently studying the textbook Neural Networks and Deep Learning by Charu C. Aggarwal. Chapter 1.2.1.3 Choice of Activation and Loss Functions says the following:

The choice of activation function is a critical part of neural network design. In the case of the perceptron, the choice of the sign activation function is motivated by the fact that a binary class label needs to be predicted. However, it is possible to have other types of situations where different target variables may be predicted. For example, if the target variable to be predicted is real, then it makes sense to use the identity activation function, and the resulting algorithm is the same as least-squares regression. If it is desirable to predict a probability of a binary class, it makes sense to use a sigmoid function for activating the output node, so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, of the dependent variable is $1$.

I've read about sigmoid functions, but it isn't clear to me how they make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, of the dependent variable is $1$. So how do sigmoid functions make it so that the prediction $\hat{y}$ indicates the probability that the observed value, $y$, of the dependent variable is $1$?

EDIT: I am specifically asking about the probability that the value is $1$ (that is, how sigmoid functions specifically check for this).

$\endgroup$
2
  • $\begingroup$ Not clear whether you are asking about the target value $1$, or how sigmoid can represent probabilities at all, or how using sigmoid will result in a meaningful probability measure in practice. $\endgroup$ Jun 11, 2021 at 11:11
  • 2
    $\begingroup$ Hi @NeilSlater , I am specifically asking about the probability that the value is $1$ (that is, how sigmoid functions specifically check for this). I have edited my question for clarity. $\endgroup$ Jun 11, 2021 at 11:19

1 Answer 1

3
$\begingroup$

I am specifically asking about the probability that the value is 1 (that is, how sigmoid functions specifically check for this).

They don't in general. In the quoted text, there is an explicit constraint that means this can be the case:

If it is desirable to predict a probability of a binary class

(emphasis mine). This means that the target value $y \in \{0,1\}$. Which in turn means that training data labels will all be $0$ or $1$.

At this point, if you pass in some calculated weighted sum of features to a sigmoid function, it will output a real value between $0.0$ and $1.0$ that can be interpreted as a probability. But without more going on, it is not linked in any way to the probability of the label being $1$.

To complete things, there is also an implicit constraint (not referred in your quoted text) that you will use a training approach that will cause this weighted sum of features to drive the sigmoid to represent a meaningful probability based on the source data. The most important part of this is to use a cost function which is minimised when $\hat{y} = \mathbb{P} \{y = 1 \mid \mathbf{x}\}$ where $\mathbf{x}$ is the input features associated with a label. The most usual cost function here would be based on binary cross-entropy loss:

$$C = -\frac{1}{N}\sum_{i=1}^{N}y_i\text{log}(\hat{y}_i) + (1-y_i)\text{log}(1 -\hat{y}_i)$$

You can show using calculus that this term is minimised when $\hat{y}$ is directly related to the frequency of $y = 1$ in the training data. Adding features $\mathbf{x}$ makes this more complex, but as a first pass to understanding what this is doing, you can simply use a list of $N$ times $0$ or $1$ outputs with no input data, and show that the sum above is minimised by a fixed $\hat{y}$ equal to the proportion of $y = 1$ – the rationale is that you would use a fixed $\hat{y}$ as your guess for each value if you had no input data about any example to go on.

One important detail of the related maths is that the activation function does not affect the target of the convergence. However, it may still affect the speed and stability of the convergence. The combination of binary cross-entropy loss and sigmoid activation is popular because the gradient calculation is simple and relatively stable.


Here is the maths worked through for a population of $N$ examples $y_i$ with no feature data (so no $\mathbf{x}_i$), $M$ of which have $y_i = 1$ and $N - M$ of which have $y_i=0$. We will guess a fixed value of $\hat{y}$ so that it minimises the mean cross-entropy loss:

$$\text{argmin}_{\hat{y}} C = \text{argmin}_{\hat{y}}-\frac{1}{N}\sum_{i=1}^{N}y_i\text{log}(\hat{y}) + (1-y_i)\text{log}(1 -\hat{y})$$

The factor of $\frac{1}{N}$ can be removed as it doesn't affect what value of $\hat{y}$ causes the minimum. In addition, we have $M$ values where $y = 1$ and $N - M$ values where $y = 0$, so we can simplify the sum.

$$= \text{argmin}_{\hat{y}}-M\text{log}(\hat{y}) - (N-M)\text{log}(1 -\hat{y})$$

To find the stationary point, take the derivative and set equal to zero (I reversed signs here since $-0 = 0$).

$$\frac{d}{d\hat{y}}[ M\text{log}(\hat{y}) + (N-M)\text{log}(1 -\hat{y})] = 0$$

$$\frac{M}{\hat{y}} - \frac{N-M}{1-\hat{y}} = 0$$

$$(1-\hat{y})M - \hat{y}(N-M) = 0$$

$$M = \hat{y}N$$

$$\hat{y} = \frac{M}{N}$$

This of course is the same as the probability that a randomly selected $i \in (1,N)$ will have $y_i = 1$.

$\endgroup$
1
  • $\begingroup$ Wow, such an amazing, illuminating answer! Thank you very for taking the time to write this. $\endgroup$ Jun 12, 2021 at 1:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .