Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

Question

In music information retrieval, one usually converts an audio signal into some kind "sequence of frequency-vectors", such as STFT or Mel-spectrogram.

I'm wondering if it is a good idea to use the transformer architecture in a self-supervised manner -- such as auto-regressive models, or BERT in NLP -- to obtain a "smarter" representation of the music than the spectrogram itself. Such smart pretrained representation could be used for further downstream tasks.

From my quick google search, I found several papers which do something similar, but -- to my surprise -- all use some kind of symbolic/discrete music representation such as scores. (For instance here or here).

My question is this:

Is it realistic to train such an unsupervised model directly on the Mel spectrogram?

The loss function would not be "log softmax of next word probability", but some kind of l2-distance between "predicted vector of spectra" and "observed vector of spectra", in the next time step.

Did someone try it?

Answer 1

The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two papers you linked) encodes performances, not scores, so they include timing and accents of real performers--but are still sequences of discrete events, not audio.

There's been some work on learning discrete representations directly from audio, such as with vector-quantized variational autoencoders (VQ-VAEs). Typically an autoregressive model is trained on top of the learned representation; Jukebox used a transformer for that. By the way, I'd highly recommend reading through the "related work" section of the Jukebox paper for an overview of work on the audio/speech synthesis task.

wav2vec is probably closest to what you're describing. They train a transformer on raw audio, self-supervised, in order to learn good representations of human speech for the speech-to-text task.

As far as training directly on spectograms, there's MelNet, a somewhat exotic RNN trained for a variety of audio synthesis tasks, including music.

Hope this helps!

Answer 2

These papers are also very close to what I meant in the question (too long for a comment).

The following references come mostly from work on speech recognition.

• Mockingjay In this work, they use an analogy of Bert architecture that is fed by Mel-spectrogram, with some audio segments "masked".
  • The model is asked to reconstruct the masked parts. To avoid the model using local smoothness of audio-data, they always mask several subsequent frames, so that reasonably long segments are being masked.
  • They evaluate on downstream "Phoneme classification tasks" using the learned features and show that these learned features are stronger than raw spectrogram, in particular if little training data is available.
• Audio Albert Same story, but they use shared weights in the transformer layers. This significantly reduces memory and computational requirements and it is shown that results are comparable with Mockingjay (at least for phoneme-classification tasks).
• Tera; another Bert variant where instead of masking, they use various "Alterations" of certain audio segments.
• This and many more references are within this project with code.