This code snippet is from here under the section named "Position embeddings".

class SinusoidalPositionEmbeddings(nn.Module):
def __init__(self, dim):
super().__init__()
self.dim = dim

def forward(self, time):
device = time.device
half_dim = self.dim // 2
embeddings = math.log(10000) / (half_dim - 1)
embeddings = torch.exp(torch.arange(half_dim, device=device) * -embeddings)
embeddings = time[:, None] * embeddings[None, :]
embeddings = torch.cat((embeddings.sin(), embeddings.cos()), dim=-1)
return embeddings


The code is implementing Positional Encoding that was introduced in Transformer model.
I don't understand why we have to use torch.exp and math.log. Is this related to scaling or non-negative issue?

In section 3.5 of this paper they suggest the formula $$PE_{(pos,2i)} = \sin \left(\frac{pos}{10000^{2i/d_{model}}}\right)$$ $$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$
It turns out to be easier (or more efficient?) to implement in code using the standard maths identities $$x = \exp \log x$$ and $$\log x^a = a \log x$$ $$\frac{pos}{10000^{2i/d_{model}}} = \exp\left(\log(pos) - \frac{2i}{d_{model}} \log(10000) \right)$$