Pattern recognition for live stream Time serie

Question

I would like to submit you a problem with which I struggle.

Suppose I have this kind of record over time in a dataframe:

fig.1

If we zoom in a bit we see such shape:

fig.2

We see that the general pattern is a increase with a pick (very high or very thin sometime even flat) follow by an almost flat part with vibration then a decrease, the we go back to zero (almost) for a time (like in the middle of fig.1) or we start an other cycle

Some have very high peak, some are more flat, some have a more longuer part before to decrease.

I have 4 classes :

• increase time - 1
• working time - 2
• decrease time - 3
• rest time (no activity) - 0

Now assume in my dataframe I have columns that tell to what class belong each point in time.

I would like to build a model that can recognize those 4 class when it see it on stream data . Imagine that our stream data is fig.1 and that we read N points (on a sliding windows) over time. What model could allow me to classify correctly each point or subpart data point in this window according to a certain pattern (hope I'm clear) Regarding the fact that in reallity I could be in rest time for a very long time or in working time a very long time also. It may also depend a lot of the sliding window, we for exemple see the beginning of the increase time on the first window then it end on the next window.

I first try to use LSTM or 1D-CNN, problem I have is that it tend to see this general pattern even when it's not present.

--- UPDATE : Chillston

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=777, shuffle=False)

# input N x T x D
X_train.shape, y_train.shape, X_test.shape, y_test.shape,

((17562, 1000, 1), (17562, 1000, 4), (4391, 1000, 1), (4391, 1000, 4))

I then pass it to 1d-CNN model, I have try many Architecture even resnet-cnn version, here's it's just more classical one.

def build_res1dcnn(n_classes):

    input_shape = (X_train.shape[1], 1)

    inputs = Input(shape = input_shape, name = 'input')

    # Stage 1
    x = Conv1D(64, kernel_size=3, strides = 2, padding = 'same', activation = 'relu', 
                       kernel_regularizer = 'l2', kernel_initializer = 'he_normal',
                       bias_regularizer = 'l2')(inputs)
    x = BatchNormalization()(x)
    x = MaxPooling1D(pool_size=(2))(x)
    x = Dropout(0.2)(x)

    x = Conv1D(128, kernel_size= 5, strides = 2, padding = 'same', activation = 'relu', 
                       kernel_regularizer = 'l2', kernel_initializer = 'he_normal',
                       bias_regularizer = 'l2')(inputs)
    x = BatchNormalization()(x)
    x = MaxPooling1D(pool_size=(2))(x)
    x = Dropout(0.2)(x)

    x = Conv1D(64, kernel_size=3, strides = 2, padding = 'same', activation = 'relu', 
                       kernel_regularizer = 'l2', kernel_initializer = 'he_normal',
                       bias_regularizer = 'l2')(inputs)
    x = BatchNormalization()(x)
    x = MaxPooling1D(pool_size=(2))(x)
    x = Dropout(0.2)(x)
    
    x = GlobalMaxPooling1D()(x)
    
    # Here I want T x n_class for a T X 1 input sequence
    outputs = []

    # https://stackoverflow.com/questions/51397484/appending-tensors-in-keras
    Ty = X_train.shape[1]
    for i in range(Ty):
        out = Dense(n_classes, activation = "softmax")(x)
        outputs.append(out)

    output = Concatenate()(outputs)
    output = Reshape([Ty, n_classes])(output)
    
    
    model = Model(inputs = inputs, outputs=output)
    
    model.compile(optimizer = Adam(learning_rate = 0.1), 
                  loss = 'categorical_crossentropy', 
                  metrics = ['categorical_accuracy'])

    
    #model.summary()
    
    return model

Answer 1

There are a number of possibilities you can consider:

simple classification

Following what you have suggested as your problem definition, you can train a model for those labels, given a hisotry of some window size (your hyper-parameter). In that case, common timeseries models can be utilized (LSTM, TCN, MCNN ). I prefer the MCNN method for it is a nice model especially when fixing a proper window-size is a challenge. It probably outperforms LSTM, so chances are you'll have better results. Moreover, LSTM must be improved with Attention mechanism, as you said the main challenge of your problem is long-term dependencies.

Kernel-methods classification

Since you already know that meaningful patterns exist in your data, you can base your classification on similarity measures. Note that this is still a classification model like the one above, but now instead of raw timepoints, you provide a similarity measure for signals (or shapelets so to speak) such as Dynamic Time Warping (DTW) and do classification using kernel methods. This still requires the window-size as one of the hyper-parameters)

Switching models and timeseries segmentation

The way you have associated those patterns to a label, I think you can assume that your problem is an example of switching state space models, where you have $k$ states, each defined by a state-space model, and you switch among those states. you can assume the switching mechanism depends on the data, or the duration of the current state, which makes it interesting for your problem, since you have the challenge of long-term dependencies. switching model can learn the expected dwelling time on states, thus it will form a prior belief, not to switch too often, if it repeatedly sees rare switches. Nice thing about this model is that you can do timeseries segmentation, which marks for you your "class" regions. the learning is a bit tricky though. for that purpose, you can checkout this paper.

conclusion

In timeseries problems, the problem definition is often very important and deciding. Sometimes you aim for an impossible problem to solve, whereas with a change of perspective, you can solve it. As an example, imagine a simple linear trend with an additive noise component. if you define your goal as "predicting the next timepoint" you might not achieve satisfying "regression" results because you're basically trying to learn an independent noise. Whereas you can instead be interested in "if the timeseries will go down or up in the next timepoint" then the expected value your label is simply the slope of the trend, which is way more doable than regression. My point is, maybe you can get better result if you think of your problem as another form of research question such as segmentation, etc.