In most of the online materials that I've read, the equations of RNNs are shown only for a single layer RNN with the output going through softmax (for a many-to-one architecture).
I am trying to find out how RNNs behave in the case of having multiple layers. Let's take two layers as an example (as shown below).
- U1 = Weights on the input going into layer 1 - shape (3,4)
- V1 = Weights on the hidden state going into layer 1 - shape (4,4)
- U2 = Weights on the input going into layer 2 - shape (4,3)
- V2 = Weights on the hidden state going in layer 2 - shape (3,3)
- W = Weights on the final output - shape (3, 5)
- X1, X2, X3 are inputs to layer 1 - shape (3,1)
Since the final output Y is in the form of probabilities, I understand that it has to go through a softmax function.
My Question is regarding the outputs of layer 1 fed into layer 2. Does the RNN apply softmax to them as well?