What's wrong with getting a dynamic RNN's output at the end of a padded sequence?

Question

Would it not even be more helpful? It will have more time to transform the input with the extra zeros padded to the end.

Answer 1

There is nothing wrong in theory. It is reasonable to expect that for some problems it will work worse than taking the immediate answer on the last known part of the sequence, some it will have roughly the same effect, and for others it might result in a better result.

A trained RNN is essentially a learned arbitrary program. If a computer program that processed arrays would get a better result by running more iterations at the end of an array, then so might a RNN. One example that comes to mind is predicting the end point of a trajectory of an observed object, when you have a variable number of observations at the start. A program that simulated each timestep in a simple fashion could well perform better than one that attempted to analyse and project directly from whatever the last observed state was.

On the other hand, you may face some practical problems:

• One practical detail of not padding sequences is that it may reduce computation time since the network is run less often.

• A padded sequence will only get a gradient to learn from with the last output, after the padding. This could make learning harder than it needs to be with vanishing or exploding gradients (these are more of a problem, the more time steps you need take back through the network in order to assign correct weights due to influence of first input).

The usual way to find out is to try it in an experiment. See which approach does better on a hold-out test set when training with or without padding. Bear in mind that whatever the answer it will not hold in general, but is likely to hold for problems similar to the one you tested with.