New answers tagged

0 votes
Accepted

Going beyond intent classification

Found it! After a lot of searching and reading through different papers, this was the exact one I needed. "BERT for Joint Intent Classification and Slot Filling" There's also an example ...
  • 101
2 votes

How does transformer models like GPT generate valid meaningful response for meaningless garbage input?

If you give a human some input that doesn't seem to convey any meaning they will probably ask you for clarification. Presumably there are a lot of examples of this in the ChatGPT training data so that ...
  • 171
2 votes

Where to find the source code for the research paper "Attention is all you need"?

You may find the code related to the paper at https://paperswithcode.com/paper/attention-is-all-you-need.
0 votes

What is the weight matrix in self-attention?

In my mind there are two weight matrices, the one you get prior to applying softmax: $$ \alpha_{i,j} = \frac{\langle q_i, k_j \rangle}{\sqrt{d}}$$ the other you get after applying the softmax: $$ \...
0 votes

Is the multi-head attention in the transformer a weighted adjacency matrix?

Short answer, yes I believe we can! Now, here's the long answer. Suppose $X \in \mathbb{R}^{d \times n}$ has as columns $X_i$ the $d$-dimensional embeddings of the $n$-tokens $x_1, x_2, ..., x_n$ from ...
4 votes

Why don't people use nonlinear activation functions after projecting the query key value in attention?

It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing linear transformations. Attention is broadly defined as a following operation ($\text{softmax}$ is ...
  • 2,248
2 votes
Accepted

Are there versions of attention that do not require a key-value pair, but just act on one input?

Self attention only acts on one input sequence. This is actually arguable the most common form of attention seen today, as popularized by transformers. In self attention, the keys and values are the ...

Top 50 recent answers are included