Given an LSTM model with 3 cells shown below, what would be the input to the left most cell c(t-1) and h(t-1)?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
Most commonly these states are set to zero, this usually works, but it can have a negative influence on the performance.
Another option is to initialize randomly, but this is not straight forward and the performance highly depends on the noise level. In this work the noise level is selected in proportion to the prediction error of the first timestep in the series.
A third option is to use learnable variables. You can initialize these randomly and the model will learn the initialization vectors by itself. This is showcased in this answer on SO
I'd say options one and three are the most straight forward and I've mostly seen the first method.
For reference, see this paper.
-
$\begingroup$ Could random initialization make sense if you don't want the LSTM to know where the sequence begins? $\endgroup$ Commented Jun 9, 2022 at 13:42
-
$\begingroup$ I think you cannot really say that in general: The space of the hidden state follows some distribution pattern/manifold that is learned. How exactly that space looks like is up to learning, so starting with the zero-vector can fall perfectly into that hidden space and does not 'appear' as something special to the LSTM, as does random initialization. However, if it is useful for the downstream task to treat the beginning as something special, the hidden space might develop in a way that allows identifying the starting token .... $\endgroup$ Commented Jun 14, 2022 at 8:28
-
$\begingroup$ ... I could be wrong but I think you don't really have a lot of influence on that. What do you think about it? $\endgroup$ Commented Jun 14, 2022 at 8:28
$\begingroup$
$\endgroup$
Your question is related to the initial states of LSTM, where c(t-1) is the cell state (memory) and h(t-1) is the previous LSTM block output.
As pointed out here, it is reasonable to assume that those are random values.
