# Why is the hyperbolic tangent with MSE better than the sigmoid with cross-entropy?

Usually, in binary classification problems, we use sigmoid as the activation function of the last layer plus the binary cross-entropy as cost function.

However, I have already experienced (more than once) that $$\tanh$$ as activation function of last layer + MSE as cost function worked slightly better for binary classification problems.

Using a binary image segmentation problem as an example, we have two scenarios:

1. sigmoid (in the last layer) + cross-entropy: the output of the network will be a probability for each pixel and we want to maximize it according to the correct class.
2. $$\tanh$$ (in the last layer) + MSE: the output of the network will be a normalized pixel value [-1, 1] and we want to make it as close as possible the original value (normalized too).

We all know the problems associated with a sigmoid (vanishing of gradients) and the benefits of the cross-entropy cost function. We also know $$\tanh$$ is slightly better than sigmoid (zero-centered and little less prone to gradient vanishing), but when we use MSE as the cost function, we are trying to minimize a completely different problem - regression instead of classification.

Why is the hyperbolic tangent ($$\tanh$$) combined with MSE more appropriate than the sigmoid combined with cross-entropy for binary classification problems? What's the intuition behind it?

In a few words, $$\tanh$$ + MSE is like sigmoid + MSE, but with labels for classes $$-1$$ and $$1$$ instead of $$0$$ and $$1$$. If you look at the shape of $$\tanh$$ function, it has the same flat tails where changes of the argument don't change the result.