I have gone through some theoretical introductions of RNN and LSTM, which do not contain any code, and they describe in fair detail what the cells do, how they apply operations like forget, sigmoid, etc.

Then I am trying to implement them with tensorflow, and even after reading the documentation, I am unable to connect the layers' API with my theoretical understanding of the operations. For example, take the following simple code:

import tensorflow as tf # tensorflow 2.5.0
inputs=tf.random.normal(shape=(32, 10, 8))
lstm = tf.keras.layers.LSTM(units=4, return_sequences=True, return_state=True)
outputs=lstm(inputs) # Call the layer, gives a list of three tensors
lstm.trainable_weights # Gives a list of three tensors 

So what exactly is the layer doing here based on the input it receives and the weights that were initialised randomly?

If I am to implement the layer's operation myself, how do I do that?

The Google and Keras documentation contain a lot of example code, but not really explanations of the internal mathematical operations. So any help in this area, or any reference that explains the mathematical operations (not in general, but what's happening in the Tensorflow layer) would be greatly appreciated.

I have the exact same question regarding RNN and GRU layers too.

  • $\begingroup$ I think what you're asking for here is too much to expect in an answer. I'd also be curious to know the details of the mathematical implementation in Tensorflow; and this is the type of stuff I'd expect documentation for. Tensorflow is open-source software, so you can find all of the details here github.com/tensorflow/tensorflow ; are you sure there isn't documentation here for what you're looking for? At worst, since it is open-source, you can look through the code yourself (which is likely correct and of high quality, since it is a popular and well-regarded open-source project). $\endgroup$ Commented Jun 10, 2021 at 6:46
  • $\begingroup$ I have to agree with @ThePointer . In fact, to find out the details of what these layers "do", you have two accessible options: The documentation and the source code. The code that implements the LSTM class is completely open as part of the tensorflow package. Perhaps the right course is to look at those two things and then bring specific additional questions back here. $\endgroup$ Commented Jun 10, 2021 at 9:35
  • $\begingroup$ Thanks. The documentation as I see it (keras.io/api/layers/recurrent_layers/simple_rnn) does not give much hint about it. So the last option is to go through the tensorflow source code. I was not expecting a complete answer here, but was wondering whether there are any other resources on the web apart from going through the source code myself. $\endgroup$
    – Della
    Commented Jun 12, 2021 at 10:15


You must log in to answer this question.