When does an RNN use the connections that help in going backward in time?

Question

Consider the following paragraph taken from chapter 10: Sequence Modeling: Recurrent and Recursive Nets of the textbook named Deep Learning by Ian Goodfellow et al mentioning the connections of RNN to go backward in time.

For the simplicity of exposition, we refer to RNNs as operating on a sequence that contains vectors $x^{(t)}$ with the time step index $t$ ranging from 1 to $\tau$. In practice, recurrent networks usually operate on minibatches of such sequences, with a different sequence lengthτfor each member of the minibatch. We have omitted the minibatch indices to simplify notation. Moreover, the time step index need not literally refer to the passage of time in the real world. Sometimes it refers only to the position in the sequence. RNNs may also be applied in two dimensions across spatial data such as images, and even when applied to data involving time, the network may have connections that go backward in time, provided that the entire sequence is observed before it is provided to the network.

The paragraph says that the RNN can go back in time if and only if the entire sequence is provided. So, I am suspecting that it happens only during the backpropagation/backward pass.

Am I true? Or is it possible for an RNN to use those connections while forward pass also?

Answer 1

The recurrent connections are also used during the forward pass. Take a look, for example, at the following equations that compute (during the forward pass) the hidden state $h_t$ of an LSTM layer.

\begin{align} f_t &= \sigma_g(W_{f} x_t + \color{red}{U_{f}} h_{t-1} + b_f) \\ i_t &= \sigma_g(W_{i} x_t + \color{red}{U_{i}} h_{t-1} + b_i) \\ o_t &= \sigma_g(W_{o} x_t + \color{red}{U_{o}} h_{t-1} + b_o) \\ \tilde{c}_t &= \sigma_c(W_{c} x_t + \color{red}{U_{c}} h_{t-1} + b_c) \\ c_t &= f_t \circ c_{t-1} + i_t \circ \tilde{c}_t \\ h_t &= o_t \circ \sigma_h(c_t) \end{align}

The letters in $\color{red}{\text{red}}$ represent the matrices associated with the $\color{red}{\text{recurrent connections}}$, so we use them in the forward pass. Of course, before training, these connections are useless, in the sense that, if we initialize them randomly, they do not initially provide any context. However, during the backward pass, we update them based on our (labeled) data, so, over time, they should capture the temporal/recurrent relations of the sequences.