Why use exponential and log in Positional Encoding of Transformer

Question

This code snippet is from here under the section named "Position embeddings".

class SinusoidalPositionEmbeddings(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.dim = dim

    def forward(self, time):
        device = time.device
        half_dim = self.dim // 2
        embeddings = math.log(10000) / (half_dim - 1)
        embeddings = torch.exp(torch.arange(half_dim, device=device) * -embeddings)
        embeddings = time[:, None] * embeddings[None, :]
        embeddings = torch.cat((embeddings.sin(), embeddings.cos()), dim=-1)
        return embeddings

The code is implementing Positional Encoding that was introduced in Transformer model.
I don't understand why we have to use torch.exp and math.log. Is this related to scaling or non-negative issue?

Many thanks ahead!

Answer 1

The current code just implements the now-standard expressions for positional embeddings given in the original transformer paper (Attention is all you need).

In section 3.5 of this paper they suggest the formula $$PE_{(pos,2i)} = \sin \left(\frac{pos}{10000^{2i/d_{model}}}\right)$$ $$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$

It turns out to be easier (or more efficient?) to implement in code using the standard maths identities $x = \exp \log x$ and $\log x^a = a \log x$ $$\frac{pos}{10000^{2i/d_{model}}} = \exp\left(\log(pos) - \frac{2i}{d_{model}} \log(10000) \right)$$

Hopefully you can now see that this corresponds to the code you have given.

But perhaps you are interested in the question as to why the positional encodings are of this fairly whacky form. My understanding of the main intuitions are that sines and cosines interact really nicely with translation due to their periodicity, and by using exponentially spaced 'frequencies' one can extract signals for interactions at a large number of different 'length scales'; see this excellent blog post and associated links for further explanation. But as I understand it, the main reason this clever idea is so widely used is because it works so well in practice.