0
$\begingroup$

I am confused as to why the sequence length is the first dimension of the input tensor for an RNN when batch size is the first dimension for any other kind of network (Linear/CNN/etc.).

This makes me think that I haven't fully grasped the concept of RNN batches. Is each independent batch a different sequence? And is the same hidden state across batches? Is the hidden state maintained between timesteps for a given sequence (for vanilla/truncated BPTT)?

$\endgroup$
0
$\begingroup$

As it says in the documentation, you can simply reverse the order of dimensions by providing the argument batch_first=True when constructing the RNN. Then, the dimensionality will be: (batch, seq, feature), i.e. batch-size times sequence length times the dimension of your input (however dimensional that may be). Then, everything is gonna work as you are used to it.

To answer the second part of your question, normally, each sequence in a batch is independent of the others (since they commonly get sampled at random). So, there is no direct dependence between any two inputs in a batch (except, of course, for the fact that they are commonly expected to stem from some underlying shared data generating process which you want to approximate by the RNN).

And a hidden state is commonly maintained per batch element, i.e. there is one hidden state per batch element (i.e. per sequence).

| improve this answer | |
$\endgroup$
  • $\begingroup$ annoyingly this can mess things up. I was doing attention type stuff where I was interested in hidden state of encoders, but later when I did linear layers I had to have batch first. I thought that solving this would be to use batch_first = True but the output dimension of hidden state was still time x batch x features, much to my annoyance. $\endgroup$ – David Ireland Jul 8 at 12:11
  • $\begingroup$ Thank you, that explains it. But what is the logic behind this change in order which, as @david points out, can be rather inconvenient? $\endgroup$ – rboz22 Jul 8 at 13:12
  • $\begingroup$ @rboz22 imo they probably added the batch_first argument before people started doing stuff where you explicitly need the hidden state, originally this was just a 'behind the scenes' thing that you would not be particularly interested in. This is of course just a guess. $\endgroup$ – David Ireland Jul 8 at 19:33
  • $\begingroup$ Also, I think it might be motivated by how real-life datasets are set up. I could imagine that there are many sets where you have for each time point a set of measurements (i.e. your batch elements) and then want to have sequence first, consequently. $\endgroup$ – Daniel B. Jul 8 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.