Why is the hyperbolic tangent with MSE better than the sigmoid with cross-entropy?

Question

Usually, in binary classification problems, we use sigmoid as the activation function of the last layer plus the binary cross-entropy as cost function.

However, I have already experienced (more than once) that $\tanh$ as activation function of last layer + MSE as cost function worked slightly better for binary classification problems.

Using a binary image segmentation problem as an example, we have two scenarios:

1. sigmoid (in the last layer) + cross-entropy: the output of the network will be a probability for each pixel and we want to maximize it according to the correct class.
2. $\tanh$ (in the last layer) + MSE: the output of the network will be a normalized pixel value [-1, 1] and we want to make it as close as possible the original value (normalized too).

We all know the problems associated with a sigmoid (vanishing of gradients) and the benefits of the cross-entropy cost function. We also know $\tanh$ is slightly better than sigmoid (zero-centered and little less prone to gradient vanishing), but when we use MSE as the cost function, we are trying to minimize a completely different problem - regression instead of classification.

Why is the hyperbolic tangent ($\tanh$) combined with MSE more appropriate than the sigmoid combined with cross-entropy for binary classification problems? What's the intuition behind it?

Answer 1

See the blog post Why You Should Use Cross-Entropy Error Instead Of Classification Error Or Mean Squared Error For Neural Network Classifier Training (2013) by James D. McCaffrey.

It should give you an intuition of why the average cross-entropy (ACE) is more appropriate than MSE (but MSE is also applicable).

In a few words, $\tanh$ + MSE is like sigmoid + MSE, but with labels for classes $-1$ and $1$ instead of $0$ and $1$. If you look at the shape of $\tanh$ function, it has the same flat tails where changes of the argument don't change the result.