3
$\begingroup$

I am working on a project that requires time-series prediction (regression) and I use LSTM network with first 1D conv layer in Keras/TF-gpu as follows:

model = Sequential()
model.add(Conv1D(filters=60, activation='relu', input_shape=(x_train.shape[1], len(features_used)), kernel_size=5, padding='causal', strides=1))
model.add(CuDNNLSTM(units=128, return_sequences=True))
model.add(CuDNNLSTM(units=128))
model.add(Dense(units=1))

As an effect my model is clearly overfitting: enter image description here

So I decided to add dropout layers, first I added layers with 0.1, 0.3 and finally 0.5 rate:

model = Sequential()
model.add(Dropout(0.5))
model.add(Conv1D(filters=60, activation='relu', input_shape=(x_train.shape[1], len(features_used)), kernel_size=5, padding='causal', strides=1))
model.add(Dropout(0.5))
model.add(CuDNNLSTM(units=128, return_sequences=True))
model.add(Dropout(0.5))
model.add(CuDNNLSTM(units=128))
model.add(Dense(units=1))

However I think that it has no effect on the network learning process, even though 0.5 is quite large dropout rate: enter image description here

Is this possible that dropout has little/no effect on a training process of LSTM or maybe I do something wrong here?

[EDIT] Adding plots of my TS, general and zoomed in view. enter image description here enter image description here

I also want to add that the time of training increases just a bit (i.e. from 1540 to 1620 seconds) when I add the dropout layers.

$\endgroup$
2
$\begingroup$

A couple of points:

  1. Have you firstly scaled your data, e.g. using MinMaxScaler? This could be one reason why your loss readings remain high.

  2. Additionally, consider that while Dropout can be useful for reducing overfitting, it is not necessarily a panacea.

Let's take an example of using LSTM to forecast fluctuations in weekly hotel cancellations.

Model without Dropout

# Generate LSTM network
model = tf.keras.Sequential()
model.add(LSTM(4, input_shape=(1, previous)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
history=model.fit(X_train, Y_train, validation_split=0.2, epochs=20, batch_size=1, verbose=2)

Over 20 epochs, the model achieves a validation loss of 0.0267 without Dropout.

loss without dropout

Model with Dropout

# Generate LSTM network
model = tf.keras.Sequential()
model.add(LSTM(4, input_shape=(1, previous)))
model.add(Dropout(0.5))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
history=model.fit(X_train, Y_train, validation_split=0.2, epochs=20, batch_size=1, verbose=2)

However, validation loss is slightly higher with Dropout at 0.0428.

loss with dropout

  1. Make sure you have specified the loss function correctly. If you are forecasting a time series, then you are most likely working with interval data. Therefore, mean_squared_error is an appropriate loss function as one is trying to estimate the deviation between the predicted and actual values.

As a counterexample, binary_crossentropy would not be suitable as the time series is not a classification set. However, misspecifying the loss function is a common error. Therefore, you also want to make sure you are using the appropriate loss function and then work from there.

| improve this answer | |
$\endgroup$
  • $\begingroup$ 1. Yes, I scale data with StandardScaler 2. This might be the answer, but why this happens? What's the remedy? 3. Yes, I use MSE as a loss function $\endgroup$ – Makintosz Jan 4 at 14:57
  • $\begingroup$ Can you please include a plot of the time series itself (if that's possible)? It will be easier for me to answer if I know the structure of the series. $\endgroup$ – Michael Grogan Jan 4 at 23:32
  • $\begingroup$ I added plots to the main question :) $\endgroup$ – Makintosz Jan 7 at 14:01
  • $\begingroup$ In your second graph, we can see a clear trend in the time series. From my experience, LSTMs don't tend to work very well with trend data. Here is an example: towardsdatascience.com/… $\endgroup$ – Michael Grogan Jan 9 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.