1
$\begingroup$

I'm having a hard time understanding how does the size of the hidden state affects GRU. For example in a concrete example lets say I want to lean a GRU to count. I'm gonna feed it fx 3 timestamps the last 3 numbers and expect it to predict the fourth. How do I know which hidden size to chose? Can I see the hidden state size as the network capabilities to encode all the past information in a fix size vector?

$\endgroup$
  • $\begingroup$ What do you mean by hidden. Are you talking about parameters, latched output of a layer that is propagated transversely through the layer (which makes it a recurrent network) or the values between the parameter weighting and the activation function? Those values that are neither input nor output are available to read in the libraries I use. Nothing is hidden except what will occur as a result of the next mini-batch. $\endgroup$ – han_nah_han_ Jan 24 at 17:45
1
$\begingroup$

Welcome to the world of deep learning! Yes, your understanding of the hidden state is correct. But the size of the hidden state is a hyperparameter that needs to found by trial-and-error. There is no closed-form formula or solution which links the size of the hidden state and the problem at hand. But, there are some rules of thumb like to start out with the size of the hidden state to be a power of 2 etc. Keep tuning the hyperparameter till you get very good predictions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.