7
$\begingroup$

Watching this video implementing attention in a transformer. He set query, key, and value biases to False and said "Typically, people don't use biases for these".

Even in official PyTorch code the default bias is False:

add_bias_kv: If specified, adds bias to the key and value sequences at dim=0. Default: False.

What is the reason behind that?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
9
$\begingroup$

For certain types of layers, such as transformers and convolutional layers, including a bias term is unnecessary and adds unnecessary overhead to the model.

The reason for this is that these layers are typically followed by a normalization layer, such as Batch Normalization or Layer Normalization. These normalization layers center the data at mean=0 (and std=1), effectively removing any bias.

Therefore, it is common practice to omit the bias term in transformers and convolutional layers that are preceded by a normalization layer.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ "preceded by a normalization layer", preceded or followed by? So, are you saying that batch normalization comes before or after the attention layer? Moreover, can you also clarify why centering the data in that way removes "biases"? You need to define bias precisely in order for that to make sense. $\endgroup$
    – nbro
    Commented Apr 30, 2023 at 20:27
  • 1
    $\begingroup$ I'm saying that the batch layer comes after the attention layer. Corrected it. As for why centering the data in that way removes "biases": x - mean(x) == (x+bias) - mean(x+bias). Same for std. The normalisation layer cancels out the bias term. $\endgroup$
    – Marc Dumon
    Commented Apr 30, 2023 at 23:18
  • 3
    $\begingroup$ How is your definition of bias related to the bias term in a layer? It seems that you're implying that the bias term in a layer would be 1. different for each $x$ (i.e. changes from input to input) and 2. the difference between each x and the mean of the distribution of the observations. What's the motivation for these implications? $\endgroup$
    – nbro
    Commented May 1, 2023 at 11:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .