1
$\begingroup$

I am working with some time-series hydrology data. Our goal is to forecast the time series forward, meaning predicting the data 1 month, 3 months ,6 months into the future. The data itself(image below) is characterized by mostly 0 or very small rates of flow expect for brief periods that are characterized by high flow. So I get this crazy spiky pattern where the median is around 0 or 1-2 meters^3/min, but at the same time there are periods of 5000 meters^3/minute, etc. I am not sure of the exact scale dimensions, but the picture below tells the tale.

enter image description here

So I was trying to figure out a good way to scale this type of spiky data. I have been using a MinMaxScaler just to start with, to rescale the values between (-1, 1). But that approach is not going to work well, especially because at the top ends of the range, the difference between 1000 m^3/min and 5000 m^3/min will be like 0.001 difference.

Does anyone have a good suggestion of how to rescale data like this for time-series analysis in an LSTM or RNN network?

Improve this question
$\endgroup$
4
  • $\begingroup$ What is your objective? prediction or detect the spikes? $\endgroup$
    – xsari3x
    Commented May 19, 2021 at 18:00
  • $\begingroup$ I want to get accurate predictions of future values. Sorry was I unclear about that? $\endgroup$
    – krishnab
    Commented May 19, 2021 at 18:02
  • $\begingroup$ And what is your input? $\endgroup$
    – xsari3x
    Commented May 21, 2021 at 13:15
  • $\begingroup$ @xsari3x our input is the image above, meaning it is the daily hydrological flow data. There are daily measurements. $\endgroup$
    – krishnab
    Commented May 21, 2021 at 13:48

1 Answer 1

Reset to default
3
$\begingroup$

First, if your data has a minimum of 0 and maximum of 5000, 1000 will get rescaled to .2 and 5000 will get rescaled to 1. So it's not a .001 difference as you suggest.

If you just used a regular loss function (e.g. mean squared error), I'm not sure you'd be able to achieve good predictions. In the literature this type of data might be called "sharp". There are some special loss functions such as DILATE (link) or soft-DTW (link) which are specially designed for time series like the one you show.

Improve this answer
$\endgroup$
1
  • $\begingroup$ oh thanks so much for pointing these papers out to me. I have seen the work of Cuturi before, especially with respect to optimal transport, but not with respect to time series. So that is excellent. I will check out these alternative loss functions. Thanks again. $\endgroup$
    – krishnab
    Commented May 21, 2021 at 19:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .