I'm experimenting with music and transformers, and I have sequences $S$ of shape: $(B,L,N)$ where $B$ is the batch size, $L$ is the sequence length, and $N=12$ are the number of musical notes with each $N_i , i\in[0,11]$ representing the probability of each note. i.e. $\sum_{i=0}^{N-1}N_i=1$

I also have embeddings $E$ for each note of shape: $(N,512)$

  1. Then I take $\text{argmax}_N (S)$ which gives most probable note of shape: $(B,L)$

  2. then pick the corresponding embedding (gather in tensorflow), resulting in shape: $(B,L,512)$ which I feed to the transformer, I'll call this $X$

  3. I also take the probability $P$ of the most probable note by taking $\text{max}_N (S)$, shape: $(B,L)$

  4. Now I feed $X$ to a transformer (or an RNN), but the problem is the transformer (or an RNN) doesnt take the probability of occurrence of that note into account.

  5. I was thinking of ways to include the probability aswell along with the embeddings, and I'm thinking of taking the direct product of $X$ and $P$ (resulting shape: $(B,L,512)$) and then feed this to the transformer, i.e. multiplying each note embedding with its occurrence probability

Question: It is okay to give the multiplied embedding ($XP$) to the transformer or an RNN?

I'm guessing it is okay because the dot products $(QK^T)$ in a transformer is simply scaled to the given probability

P.S: I'm note sure whether this question fit in AI S.E or DataScience S.E, feel free to move this question, if required.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.