This question discusses the same model mentioned in Why is the value range of my LSTM model's prediction different from my test labels?

Repeating the main points:

I am using LSTM to do time-series anomaly detection. The data is an hourly sensor input across multiple years (i.e. the global_active_energy attribute of the dataset from https://www.kaggle.com/uciml/electric-power-consumption-data-set). Data is cut into one-week sequences to predict the next 24 hours i.e. from the sequence of 01-01-2009 00:00 to 07-01-2009 23:00, predict the values of 08-01-2009 00:00 to 23:00.

The question: How should I standardise the data? In a typical problem, you use something like scikit-learn's StandardScaler, fit it around the training data, then use it to transform both the training and test data. You also use a separate StandardScaler for the training and test labels, if I recall correctly. The idea is that you shouldn't do the fitting around the test data.

In my case, however, the training data is one-week sequences preceding each day in 2008 (25-31 Dec 2007 to 24-30 Dec 2008), and the training labels are 24-hour sequences for each day in 2008.

The test data is one-week sequences from 25-31 Dec 2008 to 24-30 Dec 2009, and the test labels are 24-hour sequences for each day in 2009.

There's a lot of overlap. How should I approach this? Should I standardise the sequences, or should I standardise the original, continuous time-series before cutting them up?

If I fit StandardScaler around the training data between 25 Dec 2007 to 30 Dec 2008, then transform the test data of 25 Dec 2008 to 30 Dec 2009, should I be concerned about the overlapping dates at 25 Dec 2008 to 30 Dec 2008?

What about the labels? The training data overlaps the training labels from 1 Jan 2008 to 30 Jan 2008. Should I use the same StandardScaler in this case, or should I still use a different one?

The answer feels obvious at first glance but all this overlapping is confusing me. How should I approach this?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.