How can I use a prediction model (e.g., ARMA model or LSTM) for multi-variate data?

Question

I have a question

I have had a dataset below

             sensor1   sensor2  sensor3 ...
2021-01-01    1.32       2.2      1.0
2021-01-02    4.3        2.0      0.8 ...
...

I know ARMA model is useful for time-series forecasting

However, how can I use ARMA model for data with multiple attributes ?

If data with a single attribute can be a input for ARMA model, should I aggregate the attribute set? (for example, after normalizing each attribute, I add up all values every rows to transform all attributes to a single attribute)

Answer 1

I don't think you need to go for aggregation -- this looks like a job for VARIMA, the vector-version of ARIMA. In ARIMA, the output of the sequence at time $t$, which can be notated $X_t$, is a function of the past inputs $\{X_1, X_2, \dots, X_{t-1}\}$. For a univariate $AR(k)$ process, the corresponding ARIMA model is given by

$$ X_t - \sum_{i=1}^k \alpha_i X_{t-i} = \varepsilon_t + \sum_{i=1}^k \theta_{i}\varepsilon_{t-i}$$

with parameters $\alpha_i, \theta_i$, inputs $X_i$, and i.i.d. zero-mean Gaussian error terms $\varepsilon_i$. The generalization of this to multiple variables is thus simply

$$\mathbf{x}_t - \sum_{i=1}^k \mathbf{A}_i \mathbf{x}_{t-1} = \mathbf{e}_t + \sum_{i=1}^k \mathbf{\Theta}_i \mathbf{e}_{t-i}$$

for $\mathbf{x}_i, \mathbf{e}_i \in \mathbb{R}^n$. Note that before, the parameters were scalars. Now, $\mathbf{A}_i, \mathbf{\Theta}_i \in \mathbb{R}^{n\times n}$ -- size-$n$ square matrices.

It looks like there's a Github implementation here as well, though I haven't looked closely at this.

If you're going for an LSTM-based sequence modeling approach, this is even easier -- an LSTM cell can take in input of arbitrary dimensions, so you shouldn't have to make any changes.

If you'd like to see the math, concretely, the LSTM forward pass equations have the form

$$(\cdot)^{(t)} = g(\mathbf{W}^{(\cdot)} x^{(t)} + \mathbf{U}^{(\cdot)}x^{(t)} + \mathbf{b}^{(\cdot)})$$

where $\mathbf{W}, \mathbf{U}$ are the parameter matrices associated with the inputs and hidden states, respectively, for each gate, and $\mathbf{b}$ is a bias term. So in the single-variable case, $\mathbf{W}, \mathbf{U}, \mathbf{b}$ are all scalars; in the multivariate case, $\mathbf{W}, \mathbf{U}$ are now size-$n$ square matrices, and $\mathbf{b}$ is a vector of length $n$. No further modification is needed, and you should be able to just plug-and-play with most deep learning libraries.

So your inputs would just be the vector of attributes at a particular time step.