New answers tagged attention
0
votes
Accepted
Going beyond intent classification
Found it! After a lot of searching and reading through different papers, this was the exact one I needed.
"BERT for Joint Intent Classification and Slot Filling"
There's also an example ...
- 101
2
votes
How does transformer models like GPT generate valid meaningful response for meaningless garbage input?
If you give a human some input that doesn't seem to convey any meaning they will probably ask you for clarification. Presumably there are a lot of examples of this in the ChatGPT training data so that ...
- 171
2
votes
Where to find the source code for the research paper "Attention is all you need"?
You may find the code related to the paper at https://paperswithcode.com/paper/attention-is-all-you-need.
0
votes
What is the weight matrix in self-attention?
In my mind there are two weight matrices, the one you get prior to applying softmax:
$$ \alpha_{i,j} = \frac{\langle q_i, k_j \rangle}{\sqrt{d}}$$
the other you get after applying the softmax:
$$ \...
0
votes
Is the multi-head attention in the transformer a weighted adjacency matrix?
Short answer, yes I believe we can! Now, here's the long answer. Suppose $X \in \mathbb{R}^{d \times n}$ has as columns $X_i$ the $d$-dimensional embeddings of the $n$-tokens $x_1, x_2, ..., x_n$ from ...
4
votes
Why don't people use nonlinear activation functions after projecting the query key value in attention?
It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing linear transformations.
Attention is broadly defined as a following operation ($\text{softmax}$ is ...
- 2,248
2
votes
Accepted
Are there versions of attention that do not require a key-value pair, but just act on one input?
Self attention only acts on one input sequence. This is actually arguable the most common form of attention seen today, as popularized by transformers. In self attention, the keys and values are the ...
- 2,005
Top 50 recent answers are included