6
$\begingroup$

I was following some examples to get familiar with TensorFlow's LSTM API, but noticed that all LSTM initialization functions require only the num_units parameter, which denotes the number of hidden units in a cell.

According to what I have learned from the famous colah's blog, the cell state has nothing to do with the hidden layer, thus they could be represented in different dimensions (I think), and then we should pass at least 2 parameters denoting both #hidden and #cell_state.

So, this confuses me a lot when trying to figure out what the TensorFlow's cells do. Under the hood, are they implemented like this just for the sake of convenience or did I misunderstand something in the blog mentioned?

dimensions illustration

$\endgroup$
  • $\begingroup$ @kuixiong I have the same question, the PyTorch API also forces the hidden state and the cell state to have the same dimension, and I don't see why that has to be the case. I'm thinking perhaps it is for ease of implementation, but I don't see why that shortcut would be necessary. It seems to me that as long as you make sure the gates are already aware of the size of the concatenated [hidden,input] vector, so they can operate on it and return a vector of the same size as the cell state, then it's OK to have different hidden and cell state dimensions. $\endgroup$ – Mike Apr 10 at 22:28
0
$\begingroup$

I had a very similar issue as you did with the dimensions. Here's the rundown:

Every node you see inside the LSTM cell has the exact same output dimensions, including the cell state. Otherwise, you'll see with the forget gate and output gate, how could you possible do an element wise multiplication with the cell state? They have to have the same dimensions in order for that to work.

Using an example where n_hiddenunits = 256:

Output of forget gate: 256
Input gate: 256
Activation gate: 256
Output gate: 256
Cell state: 256
Hidden state: 256

Now this can obviously be problematic if you want the LSTM to output, say, a one hot vector of size 5. So to do this, a softmax layer is slapped onto the end of the hidden state, to convert it to the correct dimension. So just a standard FFNN with normal weights (no bias', because softmax). Now, also imagining that we input a one hot vector of size 5:

input size: 5
total input size to all gates: 256+5 = 261 (the hidden state and input are appended)
Output of forget gate: 256
Input gate: 256
Activation gate: 256
Output gate: 256
Cell state: 256
Hidden state: 256
Final output size: 5

That is the final dimensions of the cell.

| improve this answer | |
$\endgroup$
  • $\begingroup$ All the gates have to have the same dimension output as the cell state in order to perform element-wise multiplications. However, this does not answer why the hidden state and the cell state vectors have to have the same dimensions. You could very well have different hidden and cell state dimensions and still force the concatenation of the hidden state with the input vector to have the same dimension as the cell state after going through the gates. $\endgroup$ – Mike Apr 10 at 22:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.