Are positional embeddings computed during or before training?

Question

I'm trying to practically frame the concept of positional embeddings as introduced in the original paper.

As far as I've understood, what we do is basically creating some other vectors in addition to the original embeddings of our inputs. So if I have my input $X \in \mathbb{R}^{n \times d}$ with shape (64,103) (so $n$ here is the batch size), I will be creating a matrix $P \in \mathbb{R}^{n \times d}$ where each $d$ dimensional vector will contain information about positions. Now, these vectors are generated from sinusoid functions with an initial frequency of 1e-4, and then initial embeddings $X \in \mathbb{R}^{n \times d}$ and positional embeddings $P \in \mathbb{R}^{n \times d}$ are summed together to get a new input $X' \in \mathbb{R}^{n \times d}$.

Now, all this process is happening before the start of training right? Positional embeddings are not learned/modified during backpropagation?

Answer 1

Yes, your reading of the paper is correct. Vectors generated from sinusoid functions are fixed, and are not modified during training.

There exists an alternative variant -- initialize vectors as random and then update during training as usual. It also works in practice.