Can LSTM model use ReLU or LeakyReLU as the activation funtion?

Question

Can LSTM model use ReLU or LeakyReLU as the activation funtion? If so, when should one use tanh and when should one use ReLU or LeakyReLU?

Answer 1

Yes, you can use ReLU or LeakyReLU in an LSTM model.

There aren't hard rules for choosing activation functions. Run your model with each activation function and pick the best performing one.

See the following academic papers on activation functions for LSTM:

1. Towards activation function search for long short-term model network: A differential evolution based approach
2. Flood Prediction using Deep Learning Models

The following are some more articles which may be of interest although they don't specifically address LSTMs:

1. Activation Functions in Neural Networks [12 Types & Use Cases]
2. How to Choose an Activation Function for Deep Learning
3. How to Choose the Right Activation Function for Neural Networks
4. Performance Analysis of Various Activation Functions Using LSTM Neural Network For Movie Recommendation Systems
5. Activation Functions — All You Need To Know!
6. ALReLU: A different approach on Leaky ReLU activation function to improve Neural Networks Performance
7. Activation Functions Compared With Experiments

Answer 2

Yes an LSTM can use any of these.

There are no hard rules of which to use. That is why they all exist.

Some rules of thumb are:

1. Relu is the cheapest computationally. Almost always worth trying first.

2. If you find neurons are dying try leaky relu instead.

3. Tanh is probably not going to help and is much slower. Likely only worth trying if computational resources are plentiful