Usually, in binary classification problems, we use sigmoid as the activation function of the last layer plus the binary cross-entropy as cost function.
However, I have already experienced (more than once) that $\tanh$ as activation function of last layer + MSE as cost function worked slightly better for binary classification problems.
Using a binary image segmentation problem as an example, we have two scenarios:
- sigmoid (in the last layer) + cross-entropy: the output of the network will be a probability for each pixel and we want to maximize it according to the correct class.
- $\tanh$ (in the last layer) + MSE: the output of the network will be a normalized pixel value [-1, 1] and we want to make it as close as possible the original value (normalized too).
We all know the problems associated with a sigmoid (vanishing of gradients) and the benefits of the cross-entropy cost function. We also know $\tanh$ is slightly better than sigmoid (zero-centered and little less prone to gradient vanishing), but when we use MSE as the cost function, we are trying to minimize a completely different problem - regression instead of classification.
Why is the hyperbolic tangent ($\tanh$) combined with MSE more appropriate than the sigmoid combined with cross-entropy for binary classification problems? What's the intuition behind it?