Usually, in binary classification problems, we use sigmoid as the activation function of the last layer plus the binary cross-entropy as cost function.

However, I have already experienced (more than once) that $\tanh$ as activation function of last layer + MSE as cost function worked slightly better for binary classification problems.

Using a binary image segmentation problem as an example, we have two scenarios:

  1. sigmoid (in the last layer) + cross-entropy: the output of the network will be a probability for each pixel and we want to maximize it according to the correct class.
  2. $\tanh$ (in the last layer) + MSE: the output of the network will be a normalized pixel value [-1, 1] and we want to make it as close as possible the original value (normalized too).

We all know the problems associated with a sigmoid (vanishing of gradients) and the benefits of the cross-entropy cost function. We also know $\tanh$ is slightly better than sigmoid (zero-centered and little less prone to gradient vanishing), but when we use MSE as the cost function, we are trying to minimize a completely different problem - regression instead of classification.

Why is the hyperbolic tangent ($\tanh$) combined with MSE more appropriate than the sigmoid combined with cross-entropy for binary classification problems? What's the intuition behind it?


See the blog post Why You Should Use Cross-Entropy Error Instead Of Classification Error Or Mean Squared Error For Neural Network Classifier Training (2013) by James D. McCaffrey.

It should give you an intuition of why the average cross-entropy (ACE) is more appropriate than MSE (but MSE is also applicable).

In a few words, $\tanh$ + MSE is like sigmoid + MSE, but with labels for classes $-1$ and $1$ instead of $0$ and $1$. If you look at the shape of $\tanh$ function, it has the same flat tails where changes of the argument don't change the result.

| improve this answer | |
  • $\begingroup$ So, he's saying the problem with MSE is that the gradients get smaller when the predictions are close to 1 and 0, and so the network might get stuck on these outputs even when they're wrong? I don't see why cross entropy is different in this regard... He said that it applies to softmax and that the math is very cool, but didn't elaborate. Can you expand on his explanation, and also apply it to tanh? $\endgroup$ – feature_engineer Jul 27 at 6:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.