# Tag Info

• 1,400

### How to assess if OpenAI's ChatGPT chatbot has a human in the loop?

Until you can prove that OpenAI has HITL in ChatGPT, it is just an idea with no basis. It's not up to us to disprove it, it's up to you to prove it. Let me address your points: You seem to be basing ...
• 179

### Is there any artificial intelligence that possesses "concentration"?

Concentration, perhaps easier to grasp as "focus" or "attention", has quite some history in AI. This answer mentions CopyCat, and there was work with neural networks in the 80s as well (e.g. from ...
• 1,490
Accepted

• 2,222

### What kind of word embedding is used in the original transformer?

No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings. ...
• 1,678

### In layman terms, what does "attention" do in a transformer?

Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is ...
• 4,753
Accepted

### Transformers: how to get the output (keys and values) of the encoder?

I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py). It seems like an extra linear layer learns ...
• 1,400
Accepted

### Any comparison between transformer and RNN+Attention on the same dataset?

If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation ...
• 1,723

### What is the difference between Attention Gate and CNN filters?

CNNs work by applying filters over the entire image. The same filter is applied at every pixel in the image. That is, the same weights are used at every pixel. Note, when I say "at every pixel&...
Accepted

### A mathematical explanation of Attention Mechanism

There's plenty, but keep in mind that these articles do not describe the same approach. They simply have attention shifting automation as part of their approaches and therefore must detect a need for ...
• 7,375
Accepted

### What is the intuition behind the attention mechanism?

Simply put, the attention mechanism is loosely inspired on well, attention. Consider we are attempting machine translation on the following sentence: "The dog is a Labrador." If you were to ask ...

### Why does GPT-2 Exclude the Transformer Encoder?

The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot ...
• 51

### Isn't attention mask for BERT model useless?

This is just an implementation issue. One reason is the Huggingface implementation (which is not the original implementation by Google) wants to strictly separate the tokenization from the modeling. ...
• 241
Accepted

### When exactly does the split into different heads in Multi-Head-Attention occur?

The queries, keys and values are calculated then chunked so that each chunk depends on (is a linear combination of) all values of the input embedding. As for understanding an implementation, I didn't ...
Accepted

### What is the most important predecessor of the transformer model?

An influential predecessor paper is indeed the work on NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE. The paper outlines an attentional mechanism that is similar to the ...
• 373
Accepted

### What is the intuition behind self-attention?

First of all, I totally agree that the Iron Man example is a little off the topic and not really clear to explain the concept of self-attention. But she had a point there. Just like what you said, the ...
• 81