0
$\begingroup$

I am training a stateful LSTM network, with a time series consisting of about 500000 data points spread over 5 years. This time series is split up to batches of 100 timesteps, and fed into the network. The predicted signal contains artefacts every 100 timesteps, even though I expect the network to maintain state across batches and not do this "reset looking" behaviour. Any ideas of where my problem lies?

Example of artefacts. It looks like exponentially decaying spikes, making it look like some sort of state reset and then finding its way back again. Example of artefacts. It looks like exponentially decaying spikes, making it look like some sort of state reset and then finding its way back again.

How the data is split:

    samples_to_remove_from_training_set = x_train.shape[0]%num_of_steps
x_train = x_train[:-samples_to_remove_from_training_set]
y_train = y_train[:-samples_to_remove_from_training_set]
normed_train_dataset = np.reshape(x_train.to_numpy(), newshape=(-1, num_of_steps, no_of_features))
train_labels = np.reshape(y_train.to_numpy(), newshape=(-1, num_of_steps, 1))
assert normed_train_dataset.shape[0] == train_labels.shape[0]
# test set
samples_to_remove_from_test_set = x_test.shape[0]%num_of_steps
print(x_test)
x_test = x_test[:-samples_to_remove_from_test_set]
print(x_test)
y_test = y_test[:-samples_to_remove_from_test_set]
normed_test_dataset = np.reshape(x_test.to_numpy(), newshape=(-1, num_of_steps, no_of_features))
print(normed_test_dataset.shape)
test_labels = np.reshape(y_test.to_numpy(), newshape=(-1, num_of_steps, 1))
assert normed_test_dataset.shape[0] == test_labels.shape[0]

The network:

  model.add(layers.LSTM(10, activation='tanh', batch_input_shape=(batch_size, num_of_steps, no_of_features), return_sequences=True, stateful=True))
  model.add(layers.Dropout(dropout_rate))
  model.add(layers.Dense(units=10, activation='linear'))
  model.add(layers.Dropout(dropout_rate))
  model.add(layers.Dense(units=1, activation='linear'))

How predictions are made:

model.reset_states()
train_predicitons = pd.Series(np.reshape(model.predict(normed_train_dataset, batch_size=batch_size),newshape=(-1)))
model.reset_states()
predictions = pd.Series(np.reshape(model.predict(normed_test_dataset, batch_size=batch_size),newshape=(-1)))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.