# Backpropagation after N sequential input-output pass

I'm trying to train a Neural Network in a particular situation -- similar to a genetic algorithm domain as far as I know. I have to run a simulation with a length of $$K$$ steps. I have a neural network $$N$$ that at each time step is used to produce an output, so that: $$o_{t+1} = N(i_{t})$$ $$i_t$$ is a feature vector built upon $$o_{t-1}$$, and $$i_0$$ is given. My ground-truth value is $$o_k$$, namely the right value at the end of the simulation. So, I can evaluate the loss (e.g. MSE) only at the end of the simulation. Suppose to fix k to 3, the evaluation is: $$N(N(N(i_0))$$ because: $$o_1 = N(i_o) \\ o_2 = N(i_1) \\ o_3 = N(i_2)$$ So my questions are:

• does it have any sense to apply backpropagation in these settings?
• if yes, what happens to the gradients?

Practically, in some simple situations, the backpropagation seems to work, but in others, the gradients explode or vanishes

• Is it possible that you could train it in steps, starting from short simulations and then working up to longer ones? Jul 30 at 15:14
• In some case it is possible, but in the general case no.. Aug 2 at 6:52

## I/ Does it has any sense to apply backpropagation in these settings?

If I understand correctly, this question should be "Can backpropagation return the gradient for every node of these networks?".

It depends on your network $$N$$ if it is differentiable with all inputs $$i_t$$ so the backpropagation can be guaranteed that there are the gradient values at every node of the network.

## II/ If yes, what happens to the gradients?

We deal with the vanishing or exploding gradient after we make sure that the gradient is exist. The reason for this phenomenon in a neural network can come various, some typical causes are:

1. Very deep neural networks (a large number of layers):

Assume a neural network with $$n$$-th layers, the backpropagation is followed the Markov Chain's rule can be show as:

$$\frac{\partial loss}{\partial w_t} = \frac{\partial loss}{\partial l_{n}} \times \frac{\partial l_{n}}{\partial l_{n-1}}\times...\times \frac{\partial l_{t+1}}{\partial l_{t}}\times \frac{\partial l_t}{\partial w_t}$$

It's easy to see that the gradient is scaled exponentially for each layer, so if each gradient value is larger than 1, the gradient will become $$\inf$$ with $$n\rightarrow \inf$$ (exploding) and $$0$$ in vice versa.

1. Activation function:

The image below shows the difference between different activation functions. It's easy to see that the logistic function such as Sigmoid or Tanh is limited in the range $$[0,1]$$ (in case sigmoid) or $$[-1,1]$$ in case Tanh. Therefore, if the output from a node of the neural network is larger than 2 or smaller than -2, the gradient will become nearly zero (vanishing) because the output is always the same. Solution:

• Replace it by the simple function such as ReLU: $$o_{t+1} = max(0, N(i_t))$$. However, the output from ReLU will be $$0$$ if it's lower than $$0$$, so there are many variants of ReLU to solve this problem, you can find them easily by the keyword "ReLU family" on google (example).