In PyTorch, why does the sequence length need to be provided as the first dimension of the input tensor for an RNN?

I am confused as to why the sequence length is the first dimension of the input tensor for an RNN, while the batch size is the first dimension for any other kind of network (linear, CNN, etc.).

This makes me think that I haven't fully grasped the concept of RNN batches. Is each independent batch a different sequence? And is the same hidden state across batches? Is the hidden state maintained between timesteps for a given sequence (for vanilla/truncated BPTT)?

As it says in the documentation, you can simply reverse the order of dimensions by providing the argument batch_first=True when constructing the RNN. Then, the dimensionality will be: (batch, seq, feature), i.e. batch-size times sequence length times the dimension of your input (however dimensional that may be). Then, everything is gonna work as you are used to it.
• annoyingly this can mess things up. I was doing attention type stuff where I was interested in hidden state of encoders, but later when I did linear layers I had to have batch first. I thought that solving this would be to use batch_first = True but the output dimension of hidden state was still time x batch x features, much to my annoyance. – David Ireland Jul 8 '20 at 12:11
• @rboz22 imo they probably added the batch_first argument before people started doing stuff where you explicitly need the hidden state, originally this was just a 'behind the scenes' thing that you would not be particularly interested in. This is of course just a guess. – David Ireland Jul 8 '20 at 19:33