1
$\begingroup$

How come all the multi-headed self-attention layers don't end up learning the same aspect of a natural language? Since we don't dictate ahead of time what the self-attention layers focus on, how do we ensure they don't "converge"?

(My question is partly influenced by the notion of kernels in computer vision. Those kernels also focus on different aspects of the image, but they don't end up learning the same thing since they're specified ahead of time.)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

They all have different weight initializations. I think that the chance of gradient descent discovering the same local minima in such high dimensions is low.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .