Can I initialize the initial state of my RNN to be non-zero? I have some initial condition of the sequence and I want to use this initial condition as the initial state.
1 Answer
$\begingroup$
$\endgroup$
2
Can I initialize the initial state of my RNN to be non-zero? Yes, the initial state / weights of a neural network can be initialized to non-zero values. In fact, the trainable weights can be initialized any way you want.
Exactly how you do this depends on the deep learning package that you are using (if you are using a package). For example, if you are using Keras/TensorFlow, there is built-in functionality to initialize layer weights and set model weights. Here is a nice article demonstrating how this can be done: "How to set custom weights in keras using NumPy array". The article uses random normal initialization, but you can use your desired initial state / weights.
answered Oct 23, 2022 at 22:15
-
$\begingroup$ I think OP was maybe asking about the initial state, i.e. first hidden state not the initialization of the network. - @wrek? In that case, here is a related thread that might yield an answer: ai.stackexchange.com/a/35465/52211 $\endgroup$ Commented Oct 24, 2022 at 13:20
-
$\begingroup$ yes, the $h_0$, the initial hidden state. $\endgroup$– wrekCommented Oct 24, 2022 at 14:47