I am working on a problem regarding the multi-classification of multivariate time signals. So I have multiple signals and try to train an algorithm on them. My current approach is to build a neural network with LSTM-layers and it works pretty fine.

I have read that LSTMs are pretty outdated because of the transformer architectures. I found some papers about the idea to use them for signal classification (see: https://arxiv.org/abs/2103.14438). There was an example on the TensorFlow page regarding univariate signal classification (see https://keras.io/examples/timeseries/timeseries_transformer_classification/). I think it is rather a research question than a common approach for this type of problem.

To my questions:

  • Would you recommend implementing a Transformer for this type of problem? Do you think, it is a more "state of the art" approach?

  • Do you know some example projects?

  • $\begingroup$ Classifying multivariate signals is a difficult task that generally requires more sophisticated methods than the standard transformer. However, if the transformer is able to accurately learn the relationships between the different variables in the signal, it can be an effective classifier. $\endgroup$
    – Faizy
    Oct 16, 2022 at 21:08

1 Answer 1

  1. Definitely yes. I would say Transformers would work wherever LSTM works and even better. The reason is that they can attend to longer sequences as an input. In contrast to transformers, in LSTM for example after some sequence length(in your case the signal length) it would lose performance.

  2. No, I don't specifically know projects that are in this domain.But you could have a look at the architecture called Conformer (https://arxiv.org/abs/2005.08100) (I think they work on audio signals)

  • $\begingroup$ Thank you for your answer! 1) I am aware of the fact, that LSTM have their benefits. Do you have any ressources (paper,benchmarks etc.) which can confirm your statement? I just feel confirmed that it is not a common approach. Maybe I should try it and publish an paper on it ;) $\endgroup$ Jan 19, 2022 at 17:33
  • $\begingroup$ Your welcome! :) I don't remember on top of my head which paper I can show you. But the reasoning is in LSTM because things are recurrent, you need the previous input for calculation and if you do this over a long sequence this leads to vanishing gradients problem therefore decreasing the learning potential. However, because transformer attends to the sentence or input sequence at once it can handle longer sequences better. I hope this helps, if I find a benchmark on this I will share for sure. medium.com/mlearning-ai/… $\endgroup$
    – Nazmican
    Jan 20, 2022 at 13:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .