Which model should I apply on sequential data?

Question

I need to predict a binary vector given a sequential dataset meaning the current datapoint depends on its predecessors as well as (known) successors.

So, it looks something like this:

Given the sequence: X = [x_1, x_2, x_3, ..., x_N]

I want to predict: Y = [y_1, y_2, y_3, ..., y_N] with y_i \in {0, 1} a binary label

Now, the sequence X is fully known, that is: future observations in the sequence are completely known at any time.

Therefore, in contrast to normal time series data, I can also harness any x_i+1, x_i+2,... from the full X sequence for predicting y_i at any time and not only x_i-2, x_i-1, x_i etc.

Also the data X is a sequence of R^dxN vectors, i.e. N d-dimensional datapoints containing real numbers.

In my case the dimensionality of the data is d=140.

Now, what I want to predict is the following:

What is y_i given X, e.g. what is y_3 given the observations x_1, x_2, x_3, x_4, x_5?

So, eventually I need something like this:

for i in range(N):
    predict y_i given X = [x_1, x_2, x_3, ..., x_N]

This actually is a many-to-one prediction task.

Now you could use a RNN, but the problem imho is that "future" observations are not taken into account when applying an RNN.

Maybe I am wrong about this assumption.

But therefore I am asking: Which model would you suggest to use for this problem?

Answer 1

Okay, I finally ended up using a Transformer based Encoder/Decoder model.

It is a very similar structure as the NLP model described in the Attention is all you need paper.

However, since my use case is not NLP but rather sequential data in general I used to follow the Transformer Encoder/Decoder method as described in this paper: Deep Transformer Models for Time Series Forecasting

I slightly modified their model in my implementation to account for my special needs. I used pytorch to build the transformer architecture.

The results are very good.

I think when trying to predict sequential data, a encoder/decoder based transformer model will be a good choice and at least worth a look :)

Answer 2

A more recent model you can try for sequential data is the S4 model, which is a more lightweight and stable recurrent network that was able to perform better than transformer models on certain tasks. For time series forecasting in particular, there are cases where standard recurrent networks (LSTM, GRU, etc) are still preferable compared to transformer models.