How to predict time signal based on multi-input signals?

Question

I would like to approximate the following relation by a neural network

$y = \mathcal{f}(x_1(t),x_2(t))$

Here, I have only one output variable that is a function of 2 other variables which vary in time. Now, I want to be able to predict $y$, given any shape of the 2 independent variables in time. For this reason, I have a training set corresponding to different input and output signals. However, I don't know how to make the neural network understand the concept of time which is very important since I expect the solution at time $t_k$ to be influenced by the previous instants in time. For this reason, I added as input variable the time derivative as

$y = \mathcal{f}\left(x_1(t),x_2(t), \dfrac{\partial x_1 (t)}{\partial t}\right)$

This solution seems to work quite well for the fully connected neural network that I'm using. However, I would like to know if there are other ways to treat such problems where the time history is important.

Answer 1

As for using a Neural Network for a multivariate problem I suggest that you investigate an LSTM architecture, that takes into account the temporality of events.

E.g.

model = Sequential()  
model.add(LSTM(200, activation='relu', input_shape=(n_steps_in, n_features)))  
model.add(RepeatVector(n_steps_out))  
model.add(LSTM(200, activation='relu', return_sequences=True))  
model.add(TimeDistributed(Dense(n_features)))  
model.compile(optimizer='adam', loss='mse') 

I also suggest that you investigate traditional statistics models such as SARIMAX, and test for Autocorrelation and Partial Autocorrelation (the incidence of the last event, and the incidence of the events chain on your actual sample).

Answer 2

Instead of feeding the network the samples at just a single timestamp $t$, you should feed it the last $n$ observations. Since you have two "channels" $x_1$ and $x_2$, your input to the network would have dimensions $\texttt{batch_size} \times n \times 2$ (or $\texttt{batch_size} \times 2 \times n$ depending on your framework's conventions). These can then be processed by 1D convolutions, flattened and then the final output is produced by fully connected dense layers.

Alternatively you could flatten the input to $\texttt{batch_size} \times 2n$ and use only dense layers, but this is more prone to overfitting especially as $n$ grows larger.

Since $n$ is a hyperparameter, you'll need to run experiments to determine how large it should be. It is best to test it with exponential steps, like $1, 2, 4, 8, 16, ...$