I'm trying to practically frame the concept of positional embeddings as introduced in the original paper.
As far as I've understood, what we do is basically creating some other vectors in addition to the original embeddings of our inputs. So if I have my input $X \in \mathbb{R}^{n \times d}$ with shape (64,103)
(so $n$ here is the batch size), I will be creating a matrix $P \in \mathbb{R}^{n \times d}$ where each $d$ dimensional vector will contain information about positions. Now, these vectors are generated from sinusoid functions with an initial frequency of 1e-4
, and then initial embeddings $X \in \mathbb{R}^{n \times d}$ and positional embeddings $P \in \mathbb{R}^{n \times d}$ are summed together to get a new input $X' \in \mathbb{R}^{n \times d}$.
Now, all this process is happening before the start of training right? Positional embeddings are not learned/modified during backpropagation?