9
$\begingroup$

When and why would you not use Tanh?

I just replaced ReLU with Tanh and my model trains about 2x faster, reaching 90% acc within 500 steps.

While using ReLU it reached 90% acc in >1000 training steps.

I believe the reason it trained faster was due to a steeper gradient (correct me if I'm wrong).

This led me to wonder, why would we not use Tanh over ReLU for most contexts? Is it the computational complexity? Or something else?

Improve this question
$\endgroup$
1
  • $\begingroup$ What size and type of model is it? How long did training take (in terms of real time, i.e. hours/minutes)? $\endgroup$
    – The Guy with The Hat
    Commented May 13 at 20:54

1 Answer 1

Reset to default
10
$\begingroup$

Using tanh in hidden layers require careful initialization of network weights and works best with the input features normalized within the same range as output (i.e. -1 to 1). It have expensive functions to evaluate and suffers with vanishing gradient. But it converges faster because due to the zero centered output.

But tanh works good with RNNs, as RNNs suffers from vanishing gradients, and they need an activation function whose second derivative can sustain for long range before going to zero.

If you want any of these properties, use tanh.

Without any gist of architecture, it's hard to tell why your model train faster than ReLU. But generally speaking, ReLU trains faster than Tanh due to lack of expensive functions to evaluate (such as exponents)

Improve this answer
$\endgroup$
2
  • 5
    $\begingroup$ "trains faster" is a bit ambiguous in this respect. It could mean "in less steps" or "less wallclock time". Each ReLu step is slightly faster, but you probably need more steps. $\endgroup$
    – MSalters
    Commented May 13 at 14:17
  • $\begingroup$ Tanh isn't necessarily slow, the exponential function can be approximated very efficiently using this trick schraudolph.org/pubs/Schraudolph99.pdf . Also the cost of the activation function is usually fairly small compared to the cost of the matrix operations. I would standardise the input features rather than to -1/+1, especially if there is any chance of outlier values. $\endgroup$
    – Dikran Marsupial
    Commented May 14 at 11:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .