This code snippet is from here under the section named "Position embeddings".
class SinusoidalPositionEmbeddings(nn.Module): def __init__(self, dim): super().__init__() self.dim = dim def forward(self, time): device = time.device half_dim = self.dim // 2 embeddings = math.log(10000) / (half_dim - 1) embeddings = torch.exp(torch.arange(half_dim, device=device) * -embeddings) embeddings = time[:, None] * embeddings[None, :] embeddings = torch.cat((embeddings.sin(), embeddings.cos()), dim=-1) return embeddings
The code is implementing Positional Encoding that was introduced in Transformer model.
I don't understand why we have to use
math.log. Is this related to scaling or non-negative issue?
Many thanks ahead!