2
$\begingroup$

Can the recurrent neural network input come from a short-time Fourier transform? I mean the input is not from the time-series domain.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Yes you can apply RNN to any sequence of same data type. The sequence can be in space, time, or any arbitrary ordered list. The items in the sequence can have any data at all, the only requirement is that each represents that same kind of thing (if you have multiple types of thing to process as a sequence, you just need to expand the definition so that the input features can represent all types unambiguously - essentially creating a "base class" that can represent them all).

The RNN will consume the sequence as a time-based sequence, one item per time step of the RNN. However, you can think of that as the same as a processor clock for a computer . . . an RNN is essentially a trainable Turing machine, and in principle can learn to accumulate any data about the sequence it has seen, and output any function of that accumulated data. Although in practice this learning process might be too hard for our current systems, require immense amounts of data etc . . .

In your case, STFT does create a time-based sequence. Each item in the sequence is a frequency analysis for a short period of time, and each time step of the sequence represents a fixed time difference between STFT frames (the windows usually overlap a little), where frequencies in the signal may change. Typically each STFT frame is a single time step input to a RNN. You could input the frequency-domain values in fixed order (e.g. low to high frequency) one at a time into a RNN too, but that would be unusual and would make most learning tasks harder.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Are you sure about your answer? Frequency domain is not really analogous to time domain and I am not sure whether RNN's are the suitable architecture to handle such data $\endgroup$
    – user9947
    Oct 9, 2018 at 10:14
  • 2
    $\begingroup$ @DuttaA: Yes I am sure of my answer. A RNN/LSTM run over a sequence of STFT or MFCC frames is a standard audio processing architecture, used in multiple projects. If you convert a piece fully to frequency domain, then an RNN could still process it - actually that's not so useful for many tasks, but that's not a limitation of RNNs, just a limitation of the representation, there aren't so many target features that are clear from such a conversion, it's just as hard to process as a raw waveform. $\endgroup$
    – Neil Slater
    Oct 9, 2018 at 10:17
  • $\begingroup$ I was just wondering why use RNN when we have a fixed size i/p $\endgroup$
    – user9947
    Oct 9, 2018 at 10:22
  • $\begingroup$ @DuttaA: In this case we don't have a fixed sized input, but a series of them. $\endgroup$
    – Neil Slater
    Oct 9, 2018 at 10:24
  • $\begingroup$ @DuttaA: I would probably use a CNN (as you suggested in your answer) to process e.g. outputs of a spectrometer for chemical analysis, or other stable signal that could be converted to frequency domain, for exactly the issues you mention. $\endgroup$
    – Neil Slater
    Oct 9, 2018 at 10:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .