I'm trying to train a Neural Network in a particular situation -- similar to a genetic algorithm domain as far as I know. I have to run a simulation with a length of $K$ steps. I have a neural network $N$ that at each time step is used to produce an output, so that: $$ o_{t+1} = N(i_{t}) $$ $i_t$ is a feature vector built upon $o_{t-1}$, and $i_0$ is given. My ground-truth value is $o_k$, namely the right value at the end of the simulation. So, I can evaluate the loss (e.g. MSE) only at the end of the simulation. Suppose to fix k to 3, the evaluation is: $N(N(N(i_0))$ because: $$ o_1 = N(i_o) \\ o_2 = N(i_1) \\ o_3 = N(i_2) $$ So my questions are:

  • does it have any sense to apply backpropagation in these settings?
  • if yes, what happens to the gradients?

Practically, in some simple situations, the backpropagation seems to work, but in others, the gradients explode or vanishes

  • $\begingroup$ Is it possible that you could train it in steps, starting from short simulations and then working up to longer ones? $\endgroup$
    – user253751
    Jul 30 at 15:14
  • $\begingroup$ In some case it is possible, but in the general case no.. $\endgroup$ Aug 2 at 6:52

I/ Does it has any sense to apply backpropagation in these settings?

If I understand correctly, this question should be "Can backpropagation return the gradient for every node of these networks?".

It depends on your network $N$ if it is differentiable with all inputs $i_t$ so the backpropagation can be guaranteed that there are the gradient values at every node of the network.

II/ If yes, what happens to the gradients?

We deal with the vanishing or exploding gradient after we make sure that the gradient is exist. The reason for this phenomenon in a neural network can come various, some typical causes are:

  1. Very deep neural networks (a large number of layers):

    Assume a neural network with $n$-th layers, the backpropagation is followed the Markov Chain's rule can be show as:

$$\frac{\partial loss}{\partial w_t} = \frac{\partial loss}{\partial l_{n}} \times \frac{\partial l_{n}}{\partial l_{n-1}}\times...\times \frac{\partial l_{t+1}}{\partial l_{t}}\times \frac{\partial l_t}{\partial w_t}$$

It's easy to see that the gradient is scaled exponentially for each layer, so if each gradient value is larger than 1, the gradient will become $\inf$ with $n\rightarrow \inf$ (exploding) and $0$ in vice versa.

  1. Activation function:

    The image below shows the difference between different activation functions. It's easy to see that the logistic function such as Sigmoid or Tanh is limited in the range $[0,1]$ (in case sigmoid) or $[-1,1]$ in case Tanh. Therefore, if the output from a node of the neural network is larger than 2 or smaller than -2, the gradient will become nearly zero (vanishing) because the output is always the same.

activation function


  • Replace it by the simple function such as ReLU: $o_{t+1} = max(0, N(i_t))$. However, the output from ReLU will be $0$ if it's lower than $0$, so there are many variants of ReLU to solve this problem, you can find them easily by the keyword "ReLU family" on google (example).

There are other methods or strategies to duel with vanishing/exploding gradient, it also depends on your model or your data. The answer can be more detailed if you give more information.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.