Which positional encoding BERT use?

Question

It is a little bit confusing that someone is explaining that BERT is using sinusoidal functions for BERT position encoding and someone is saying BERT just uses absolute position.

I checked that Vaswani 2017 et al., used a sinusoidal function for position embedding, but not sure BERT used the same technique since the BERT paper does not mention it much.

My question is

1. Did BERT use absolute position embedding that is learnable, not with sinusoidal function?
2. If so, is the absolute position the same dimension as the input embedding dimension for an addition?

Answer 1

BERT uses trained position embeddings. The original paper does not say it explicitly, the term position embeddings (as opposed to encoding) suggests it is trained. When you look at BERT layers in HuggingFace Transformers, you will the dimension of the trained positions embeddings (768×512), which is also the reason why BERT cannot accept input longer than 512 tokens.

from transformers import AutoModel
model = AutoModel.from_pretrained("bert-base-cased")
print(model.embeddings)

BertEmbeddings(
  (word_embeddings): Embedding(28996, 768, padding_idx=0)
  (position_embeddings): Embedding(512, 768)
  (token_type_embeddings): Embedding(2, 768)
  (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
  (dropout): Dropout(p=0.1, inplace=False)
)