New answers tagged attention
0
The answer above is very concise but I will try to give an ELI5 example. I also agree with @nbro that attention does not exclusively mean transformer architecture.
Before attention
What is the height of the youngest female child of the father of your mother's first cousin? That query is convoluted, depends on your good memory of your family tree's ...
2
Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is that attention should be used when a model is capable of learning, or focusing on local vs global patterns in the data we use for training. And with "...
1
This is just an implementation issue. One reason is the Huggingface implementation (which is not the original implementation by Google) wants to strictly separate the tokenization from the modeling. It is a convention that the input sequences are zero-padded, but in theory, it does not have to be so. In the Huggingface implementation, you use a different ...
Top 50 recent answers are included