4
$\begingroup$

I know that when using Sigmoid, you only need 1 output neuron (binary classification) and for Softmax - it's 2 neurons (multiclass classification). But for performance improvement (if there is one), is there any difference which of these 2 approaches works better, or when would you recommend using one over the other. Or maybe there are certain situations when using one of these is better than the other. Any comments or shared experience will be appreciated.

$\endgroup$
2
1
$\begingroup$

Sigmoid is used for binary cases and softmax is its generalized version for multiple classes. But, essentially what they do is over exaggerate the distances between the various values.

If you have values on a unit sphere, apply sigmoid or softmax on those values would lead to the points going to the poles of the sphere.

$\endgroup$
1
$\begingroup$

I commonly use softmax for all 2-class or k-class problems, basically, because I always like to have an output node for each class. For sigmoid, i.e., logistic, you can estimate MSE for each sample using the relationship

$E_i = \sum_c^C (y_c - \hat{y}_c)^2$,

where $C$ is the number of classes, $y_c$ is 0 or 1 for true class membership, and $\hat{y}_c$ is the predicted class membership of the $i$th object. For example, the target or outcome truth for an object in class 2 of a 4-class problem could be $y_i=(0,1,0,0)$ while the predicted output at each of the 4 nodes could be $\hat{y}_i=(0.01,0.2,0.5,0.001)$, suggesting the predicted class is 3 (greatest probability from softmax).

During the particular training epoch, calculate $MSE$ for all the training objects as

$MSE = \frac{1}{n}\sum_i^n E_i$

Obviously, use of cross-entropy is a variation on a theme.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.