1
$\begingroup$

I am working with some time-series hydrology data. Our goal is to forecast the time series forward, meaning predicting the data 1 month, 3 months ,6 months into the future. The data itself(image below) is characterized by mostly 0 or very small rates of flow expect for brief periods that are characterized by high flow. So I get this crazy spiky pattern where the median is around 0 or 1-2 meters^3/min, but at the same time there are periods of 5000 meters^3/minute, etc. I am not sure of the exact scale dimensions, but the picture below tells the tale.

enter image description here

So I was trying to figure out a good way to scale this type of spiky data. I have been using a MinMaxScaler just to start with, to rescale the values between (-1, 1). But that approach is not going to work well, especially because at the top ends of the range, the difference between 1000 m^3/min and 5000 m^3/min will be like 0.001 difference.

Does anyone have a good suggestion of how to rescale data like this for time-series analysis in an LSTM or RNN network?

$\endgroup$
4
  • $\begingroup$ What is your objective? prediction or detect the spikes? $\endgroup$
    – xsari3x
    May 19 at 18:00
  • $\begingroup$ I want to get accurate predictions of future values. Sorry was I unclear about that? $\endgroup$
    – krishnab
    May 19 at 18:02
  • $\begingroup$ And what is your input? $\endgroup$
    – xsari3x
    May 21 at 13:15
  • $\begingroup$ @xsari3x our input is the image above, meaning it is the daily hydrological flow data. There are daily measurements. $\endgroup$
    – krishnab
    May 21 at 13:48
1
$\begingroup$

First, if your data has a minimum of 0 and maximum of 5000, 1000 will get rescaled to .2 and 5000 will get rescaled to 1. So it's not a .001 difference as you suggest.

If you just used a regular loss function (e.g. mean squared error), I'm not sure you'd be able to achieve good predictions. In the literature this type of data might be called "sharp". There are some special loss functions such as DILATE (link) or soft-DTW (link) which are specially designed for time series like the one you show.

$\endgroup$
1
  • $\begingroup$ oh thanks so much for pointing these papers out to me. I have seen the work of Cuturi before, especially with respect to optimal transport, but not with respect to time series. So that is excellent. I will check out these alternative loss functions. Thanks again. $\endgroup$
    – krishnab
    May 21 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.