Can the recurrent neural network input come from short time fourier transform in MATLAB? I mean the input is not from time series domain.


Yes you can apply RNN to any sequence of same data type. The sequence can be in space, time, or any arbitrary ordered list. The items in the sequence can have any data at all, the only requirement is that each represents that same kind of thing (if you have multiple types of thing to process as a sequence, you just need to expand the definition so that the input features can represent all types unambiguously - essentially creating a "base class" that can represent them all).

The RNN will consume the sequence as a time-based sequence, one item per time step of the RNN. However, you can think of that as the same as a processor clock for a computer . . . an RNN is essentially a trainable Turing machine, and in principle can learn to accumulate any data about the sequence it has seen, and output any function of that accumulated data. Although in practice this learning process might be too hard for our current systems, require immense amounts of data etc . . .

In your case, STFT does create a time-based sequence. Each item in the sequence is a frequency analysis for a short period of time, and each time step of the sequence represents a fixed time difference between STFT frames (the windows usually overlap a little), where frequencies in the signal may change. Typically each STFT frame is a single time step input to a RNN. You could input the frequency-domain values in fixed order (e.g. low to high frequency) one at a time into a RNN too, but that would be unusual and would make most learning tasks harder.

| improve this answer | |
  • $\begingroup$ Are you sure about your answer? Frequency domain is not really analogous to time domain and I am not sure whether RNN's are the suitable architecture to handle such data $\endgroup$ – DuttaA Oct 9 '18 at 10:14
  • 2
    $\begingroup$ @DuttaA: Yes I am sure of my answer. A RNN/LSTM run over a sequence of STFT or MFCC frames is a standard audio processing architecture, used in multiple projects. If you convert a piece fully to frequency domain, then an RNN could still process it - actually that's not so useful for many tasks, but that's not a limitation of RNNs, just a limitation of the representation, there aren't so many target features that are clear from such a conversion, it's just as hard to process as a raw waveform. $\endgroup$ – Neil Slater Oct 9 '18 at 10:17
  • $\begingroup$ I was just wondering why use RNN when we have a fixed size i/p $\endgroup$ – DuttaA Oct 9 '18 at 10:22
  • $\begingroup$ @DuttaA: In this case we don't have a fixed sized input, but a series of them. $\endgroup$ – Neil Slater Oct 9 '18 at 10:24
  • $\begingroup$ @DuttaA: I would probably use a CNN (as you suggested in your answer) to process e.g. outputs of a spectrometer for chemical analysis, or other stable signal that could be converted to frequency domain, for exactly the issues you mention. $\endgroup$ – Neil Slater Oct 9 '18 at 10:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.