I have just dived into deep learning for NLP, and now I'm learning how the BERT model works. What I found odd is why the BERT model needs to have an attention mask. As clearly shown in this tutorial https://huggingface.co/transformers/glossary.html:

from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-cased")

sequence_a = "This is a short sequence."
sequence_b = "This is a rather long sequence. It is at least longer than the sequence A."

encoded_sequence_a = tokenizer(sequence_a)["input_ids"]
encoded_sequence_b = tokenizer(sequence_b)["input_ids"]

padded_sequences = tokenizer([sequence_a, sequence_b], padding=True)

Output of padded sequences input ids:


[[101, 1188, 1110, 170, 1603, 4954, 119, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [101, 1188, 1110, 170, 1897, 1263, 4954, 119, 1135, 1110, 1120, 1655, 2039, 1190, 1103, 4954, 138, 119, 102]]

Output of padded sequence attention mask:

[[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]

In the tutorial, it clearly states that an attention mask is needed to tell the model (BERT) which input ids need to be attended and which not (if an element in attention mask is 1 then the model will pay attention to that index, if it is 0 then model will not pay attention).

The thing I don't get is: why does BERT have an attention mask in the first place? Doesn't model need only input ids because you can clearly see that attention_mask has zeros on the same indices as the input_ids. Why does the model need to have an additional layer of difficulty added?

I know that BERT was created in google's "super duper laboratories", so I think the creators had something in their minds and had a strong reason for creating an attention mask as a part of the input.


1 Answer 1


This is just an implementation issue. One reason is the Huggingface implementation (which is not the original implementation by Google) wants to strictly separate the tokenization from the modeling. It is a convention that the input sequences are zero-padded, but in theory, it does not have to be so. In the Huggingface implementation, you use a different tokenizer that would pad the sequences with different numbers and still get valid masking.

You are right that you can infer the mask from the input IDs at the very beginning (if you know the pad ID), but you need to explicitly use the mask in every single layer. Each layer returns a 3D tensor of floats from which you cannot say what the padded positions are, you need to have the explicit mask when calling the next layer. I guess that having the mask everywhere makes the API more consistent.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .