# dimensions of hidden layer and cell state layer in LSTM

I was following some examples to get familiar with tensorflow LSTM related api, but noticed that all LSTM initialization functions require only num_units parameter which denotes number of hidden units in a cell. According to what I have learnt from the famous colah's blog, cell state has nothing to do with hidden layer, thus they could be represented in different dimensions IMO, and then we should pass at least 2 parameters denoting both #hidden and #cell_state. So this confuses me a lot when trying to figure out what the tf cells do under the hood, are they implemented like this just for the sake of convenience or did I misunderstand something in the blog mentioned?

• Hi and welcome to this community! To clarify, is your question: what is the relationship between the size of a cell and "dimension" of a layer? – nbro Sep 25 at 15:50
• I am not sure what you mean by cell and layer, but according to the picture illustrated above (comes from the blog I mentioned), what I meant was the relationship between the size of the hidden($h_t$) layer and the size of the cell state layer ($C_t$). – kuixiong Sep 26 at 2:01

I had a very similar issue as you did with the dimensions. Here's the rundown:

Every node you see inside the LSTM cell has the exact same output dimensions, including the cell state. Otherwise, you'll see with the forget gate and output gate, how could you possible do an element wise multiplication with the cell state? They have to have the same dimensions in order for that to work.

Using an example where n_hiddenunits = 256:

Output of forget gate: 256
Input gate: 256
Activation gate: 256
Output gate: 256
Cell state: 256
Hidden state: 256


Now this can obviously be problematic if you want the LSTM to output, say, a one hot vector of size 5. So to do this, a softmax layer is slapped onto the end of the hidden state, to convert it to the correct dimension. So just a standard FFNN with normal weights (no bias', because softmax). Now, also imagining that we input a one hot vector of size 5:

input size: 5
total input size to all gates: 256+5 = 261 (the hidden state and input are appended)
Output of forget gate: 256
Input gate: 256
Activation gate: 256
Output gate: 256
Cell state: 256
Hidden state: 256
Final output size: 5


That is the final dimensions of the cell.