I have been experimenting with activation functions on CNN, and it occurred to me to use a rectified tanh function. So that is basically if z > 0 tanh(z) else 0. I have implemented it and I compared with ReLu on odd MNIST. They both achieved about 94% success rate in 10 epochs. My logic was that usually humans tend to stop feeling more confident once they have learnt something. Similarly, I thought a Convolutional layer neuron should not feel much more confident (higher activation) with growing evidence (higher weighted input). So is there any evidence of perhaps such a rectified tanh being more successful ?

  • $\begingroup$ I don't have any evidence, but I would say probably not. Ultimately you're trying to fit a function, and a ReLU does it so fast because of how easy it is to combine ReLU functions and also how simple it is to compute (in the forward and backward pass). I don't think you'll have much luck getting much more accuracy using a rectified tanh, but if you compare convergence time or computation time I think you'll find that a simple ReLU outperforms by quite a margin. $\endgroup$
    – Recessive
    Commented Jul 16, 2021 at 5:34
  • $\begingroup$ Yes I agree. I should mention that mathematically, rectified tanh would make it harder for the neuron to change its parameters as they grow larger (if they start out positive). This appears a useful characteristics as long you have small parameter initialization. I think it would allow you to get the extra percent or two in accuracy. For computation, perhaps a rectified arctan would perform better since its derivative is simpler than tanh. $\endgroup$
    – Physics
    Commented Jul 17, 2021 at 3:12


You must log in to answer this question.

Browse other questions tagged .