How should we regularize an LSTM model?

Question

There are five parameters from an LSTM layer for regularization if I am correct.

To deal with overfitting, I would start with

1. reducing the layers
2. reducing the hidden units
3. Applying dropout or regularizers.

There are kernel_regularizer, recurrent_regularizer, bias_regularizer, activity_regularizer, dropout and recurrent_dropout.

They have their definitions on the Keras's website, but can anyone share more experiences on how to reduce overfitting?

And how are these five parameters used? For example, which parameters are most frequently used and what kind of value should be input? ?

Answer 1

One LSTM layer should be enough unless you have lots of data. The same thing goes for the number of nodes in the layer. Start small first so 5 to 10 nodes and increment it until the performance is reasonable.

Once you have a model working you can apply regularization if you think it will improve performance by reducing overfitting of the training data. You can check this by looking at the learning curves or compring the error on the validation and test sets.

In my experiments I've used the L1 and L2 regularizers along with dropout. These can all be mixed together in fact using both L1 and L2 at the same time is called the ElasticNet.

I tend to apply the regularizers on the kernel_regularizer because this affects the weights for the inputs. Basically feature selection.

The value for the L1 and L2 can start with the default (for tensorflow) of 0.01 and change it as you see fit or read what other research papers have done.

Dropout can start at 0.1 then increment it until there is no performance gain. This is basically a percentage so 0.1 would remove about 10% of your nodes.

Finding the best regularizer is the same as any other hyperparameter optimization which is mostly trial and error.

Answer 2

If what is mentioned above, that is probably in the context of lstm networks. I would suggest using the keras tuner bayesian optimizer and making the l1 or l2 number a parameter of the kernel space. This way you find the optimal values, and its a great way to hypertune. Just keep in mind, the greater the range of parameters, or kernel if i am not wrong, the higher computer power you need.

from tensorflow import keras
import keras_tuner as kt

def model1(hp):
  model=Sequential()
  model.add(keras.layers.LSTM(units=hp.Int('units',min_value=40, max_value=800, step=20),
                              dropout=hp.Float('droput',min_value=0.15, max_value=0.99, step=0.05),
                              recurrent_dropout=hp.Float('redroput',min_value=0.05, max_value=0.99, step=0.05),
                              activation='relu',
                              return_sequences=True,
                              input_shape=(30,1)))
  Attention()
  model.add(keras.layers.LSTM(units=hp.Int('units',min_value=40, max_value=800, step=20),
                              dropout=hp.Float('droput',min_value=0.15, max_value=0.99, step=0.05),
                              activation='relu',return_sequences=True))
  Attention()
  model.add(keras.layers.LSTM(units=hp.Int('units',min_value=40, max_value=800, step=20), activation='relu'))
  model.add(keras.layers.Dense(1))
  
  model.compile(loss='mean_squared_error',optimizer=tf.keras.optimizers.Adam(hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4, 1e-7, 1e-10])))
  return model

bayesian_opt_tuner = kt.BayesianOptimization(
    model1,
    objective='val_loss',
    max_trials=200,
    executions_per_trial=1,
    project_name='timeseries_bayes_opt_POC',
    overwrite=True,)

xval=X_test
bayesian_opt_tuner.search(x=X_train ,y=X_train, 
             epochs=300,
             #validation_data=(xval ,xval),
             validation_split=0.95,
             validation_steps=30,  
             steps_per_epoch=30,
             callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_loss', 
                              patience=4,
                              verbose=1,
                              restore_best_weights=True),
                        tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', 
                                   factor=0.1, 
                                   patience= 2, 
                                   verbose=1, 
                                   min_delta=1e-5, 
                                   mode='min')]
             )
This is where the magic happens. Something I composed myself. If interested holla 

Answer 3

Regularization is trying to discourage complex information from being learned so we want to eliminate the model from actually learning to memorize the training data. We don't want to learn like very specific pinpoints of the training data that don't generalize well to test data.

Dropout, the idea of drop out is that during training we randomly set some of the activations of the hidden neurons to zero with some probability say 0.5. This idea is extremely powerful because it allows the network to lower its capacity, it also makes it such that the network can't build these memorization channels through the network where it tries to just remember the data because on every iteration 50% of that data is going to be wiped out so it's going to be forced to not only generalize better but it's going to be forced to have multiple channels through the network and build a more robust representation of its prediction. We repeat this on every iteration so on the first iteration we dropped out one 50% of the nodes on the next iteration we can drop out a different randomly sampled 50% which may include some of the previously sampled nodes as well and this will allow the network to generalize better to new test data.

Early stopping, when the network is improving its performance during training there comes a point where the training data starts to diverge from the testing data, at some point the network is going to start to do better on its training data than its testing data, what this means is basically that the network is starting to memorize some of the training data and that's what you don't want so what we can do is we can identify this inflection point where the test data starts to increase and diverge from the training data so we can stop the network early and make sure that our test accuracy is as minimum as possible.