Is there any rule of thumb for choosing the number of hidden units in an LSTM? Is it similar to hidden neurons in a regular feedforward neural network? I'm getting better results with my LSTM when I have a much bigger amount of hidden units (like 300 Hidden units for a problem with 14 inputs and 5 outputs), is it normal that hidden units in an LSTM are usually much more than hidden neurons in a feedforward ANN? or am I just greatly overfitting my problem?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ When you say "I'm getting better results with my LSTM", you need to be more precise for us to understand whether you're over-fitting or not. Can you provide more info about 1. how you're measuring the performance of your model, 2. your training dataset (i.e. how many training data points do you have?), 3. which task you're trying to solve, 4. which loss you're using, 4. how you're splitting the dataset into training, validation and test datasets, if at all, etc. Having said that, your question is partially a duplicate of ai.stackexchange.com/q/3156/2444. $\endgroup$– nbroMay 5, 2022 at 9:58
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I'm not sure about what you are referring to when you say "number of hidden units", but I will assume that it's the dimension of the hidden vector $h_t \in \mathbb{R}^N$ in this definition of an LSTM.
In general, the larger your model, in your case $N$, the more capacity your model has and, therefore, the more complex a function it can represent.
If by "better result" you mean smaller loss on the training dataset, it's very likely that you will indeed overfit more the more you increase $N$.
However, there are many techniques to increase your model expressiveness without overfitting, such as dropout.
answered May 5, 2022 at 18:59