I'm attempting a music transcription task - similar to speech recognition but with music and notes (string representations) instead of speech audio and sentences. The model consists of a CNN audio encoder and a vanilla transformer in PyTorch.

In order to batch my data, I have to pad my audio (and labels) to the same length. This is where I'm struggling to construct a padding mask for audio, especially when it needs to go through an embedding process before being fed into transformer's encoder layers.

My first intuition is to simply pad the audio with zeros to the longest length within each batch. The transformer in PyTorch expects a source key padding mask of shape (N, S) for batched input, where N is the batch size and S is the source sequence length. Let's say that a batch of audio is of shape (N, T), which would be passed through the CNN frontend and added sinusoidal positional encoding, resulting in a shape of (N, T', n_state), where T' < T depending on the stride used. Does anyone have any pointers on how create a mask for the padded audio?



You must log in to answer this question.