When and why would you not use Tanh?

I just replaced ReLU with Tanh and my model trains about 2x faster, reaching 90% acc within 500 steps.

While using ReLU it reached 90% acc in >1000 training steps.

I believe the reason it trained faster was due to a steeper gradient (correct me if I'm wrong).

This led me to wonder, why would we not use Tanh over ReLU for most contexts? Is it the computational complexity? Or something else?