# RNNs - Is the recurrence at the layer-level or at the network-level?

I am confused by where the recurrence happens in RNNs, especially in the context of deep neural networks. I am trying to transform an ordinary neural network into a recurrent one from scratch.

nn = NeuralNetwork()
nn.add(Layer(2, 2)) # fully connected layer


Does the recurrence happen at the last hidden layer of the network or after each fully connected layer?

I have heard of stacking RNNs, but I am not sure why we can't simply add a layer to an existing RNN. Is it because the recurrence is after the activation function/layer, hence the following code won't work?

import numpy as np

class Layer():
'''
Fullly connected layer
'''

def __init__(self, input_size, output_size, randomize=True):
# initialize weights and biases
if randomize:
self.W = np.random.randn(input_size, output_size)
else:
self.W = np.ones((input_size, output_size))
self.b = np.zeros((1, output_size))

def forward(self, X):
return np.dot(X, self.W) + self.b

class NeuralNetwork():
'''
Neural network divided into layers
'''

def __init__(self):
self.layers = []

'''
'''
self.layers.append(layer)

def forward(self, X):
'''
Propagate input forward through each layer
'''
for layer in self.layers:
# output of current layer becomes input of next layer
X = layer.forward(X)
return X

class ReLU():
'''
ReLU activation layer
'''

def forward(self, X):
self.X = X
return np.maximum(0, X)

class RecurrentLayer():

def __init__(self, input_size, output_size, randomize=True):
self.layer = Layer(input_size, output_size, randomize)
if randomize:
self.V = np.random.randn(output_size, output_size)
else:
self.V = np.ones((output_size, output_size))

def forward(self, X_seq):
hidden_state = []
for x_t in X_seq:
h_t = self.layer.forward(x_t)
try:
h_t += np.dot(hidden_state[-1], self.V)
except:
pass
hidden_state.append(h_t)
return hidden_state

X_sequence = [[1, 1], [2, 2], [3, 3]]
nn = NeuralNetwork()

nn.forward(X_sequence)

[array([[4.]]), array([[20.]]), array([[64.]])]


According to this paper - which discusses RNNs, LSTMs, Stacked RNNs, and even Bidirectional RNNs - a one-layer RNN is built as follows (eq 1 and 2): \begin{align} h_t &= \mathcal H(W_{xh} x_t + W_{hh}h_{t-1}+b_h)\\ y_t &= W_{hy}h_t + b_y \end{align} These two lines should be iterated for the sequence length $$T$$, so you get as output hidden states $$h=(h_1,...,h_T)$$ and sequences $$y=(y_1,...,y_T)$$. $$\mathcal{H}$$ is the activation function.
This means that the recurrence is at the level of computing the hidden state, $$h_t$$, meaning that if you have $$N$$ recurrent layers you compute all the $$h^n$$ sequences iteratively, and then when at the end you compute the output sequence $$y$$ which is also the network's output. See Eq. 11 and 12.
In general, I've seen that RNNs have different implementations. For example, in Keras the recurrence occurs on the output sequence $$y$$: so you compute all the output sequences, and optionally also return the states (if so these will inizialize the states of the next layer), which are fed to the next recurrent layer.