72 votes
Accepted

How does an AI like ChatGPT answer a question in a subject which it may not know?

ChatGPT is a large language model. That means it's very good at stringing together words in ways that humans tend to use them. It's able to construct sentences that are grammatically correct and sound ...
Mithical's user avatar
  • 2,905
71 votes
Accepted

How can neural networks deal with varying input sizes?

Three possibilities come to mind. The easiest is the zero-padding. Basically, you take a rather big input size and just add zeroes if your concrete input is too small. Of course, this is pretty ...
BlindKungFuMaster's user avatar
61 votes
Accepted

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then ...
Edoardo Guerriero's user avatar
50 votes

Why does ChatGPT fail in playing "20 questions"?

Like any other question on why ChatGPT can't do something, the simple/superficial answer is that ChatGPT is just a language model fine-tuned with RL to be verbose and nice (or to answer like the human ...
nbro's user avatar
  • 40.5k
35 votes
Accepted

Can BERT be used for sentence generating tasks?

For newbies, NO. Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its ...
soloice's user avatar
  • 511
31 votes
Accepted

How can Transformers handle arbitrary length input?

Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage. However, if you are ...
tsu's user avatar
  • 471
31 votes
Accepted

How does ChatGPT retain the context of previous questions?

Based on an answer by OpenAI, it appears that the illusion of understanding context is created by the model's capacity to accept very long input sequences. The OpenAI FAQ states that approximately ...
milez's user avatar
  • 981
30 votes

What is the intuition behind the dot product attention?

Let's start with a bit of notation and a couple of important clarifications. $\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word. $\...
Edoardo Guerriero's user avatar
29 votes
Accepted

How is BERT different from the original transformer architecture?

What is a transformer? The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-...
nbro's user avatar
  • 40.5k
27 votes

Why is ChatGPT bad at math?

chatGPT is able to create well-formed sentences which contain phrases that are fitting for the input. It has rules extracted from its data, but those are not rules of understanding, but rules of '...
gctwnl's user avatar
  • 371
25 votes

What are the main differences between skip-gram and continuous bag of words?

So as you're probably already aware of, CBOW and Skip-gram are just mirrored versions of each other. CBOW is trained to predict a single word from a fixed window size of context words, whereas Skip-...
Edoardo Guerriero's user avatar
23 votes

What is the actual quality of machine translations?

Who claimed that machine translation is as good as a human translator? For me, as a professional translator who makes his living on translation for 35 years now, MT means that my daily production of ...
Herbert's user avatar
  • 231
22 votes
Accepted

What language is the GPT-3 engine written in?

The GPT-3 source code hasn’t been released but the creators say it uses the “same model and architecture as GPT-2” (source) with some exceptions. The GPT-2 source code is written in 100% Python. The ...
S2673's user avatar
  • 590
21 votes

How can neural networks deal with varying input sizes?

Others already mentioned: zero padding RNN recursive NN so I will add another possibility: using convolutions different number of times depending on the size of input. Here is an excellent book ...
Salvador Dali's user avatar
20 votes

How does an AI like ChatGPT answer a question in a subject which it may not know?

ChatGPT does not actually know anything. But more importantly even, it does not know this fact! Hence, it does not know that it does not know. It is only good at combining text.
DrCommando's user avatar
18 votes
Accepted

What kind of word embedding is used in the original transformer?

I have found a good answer in this blog post The Transformer: Attention Is All You Need: we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some ...
Bert Gayus's user avatar
16 votes

Why does GPT-2 Exclude the Transformer Encoder?

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
Faizy's user avatar
  • 1,114
15 votes
Accepted

How would an AI learn language?

The general research area is known as grammar induction. It is generally framed as a supervised learning problem, with the input presented as raw text, and the desired output the corresponding parse ...
NietzscheanAI's user avatar
15 votes

How does an AI like ChatGPT answer a question in a subject which it may not know?

ChatGPT and other GPT-based machine learning models don't actually know anything in the sense you're thinking of. ChatGPT is a distant descendant of Markov chain text generators such as Dissociated ...
Mark's user avatar
  • 420
14 votes
Accepted

Which tasks are called as downstream tasks?

In the context of self-supervised learning (which is also used in NLP), a downstream task is the task that you actually want to solve. This definition makes sense if you're familiar with transfer ...
nbro's user avatar
  • 40.5k
14 votes

Why does ChatGPT fail in playing "20 questions"?

It Wasn't Trained To A learning system performs best on the task for which it is given explicit feedback. That is the only time the parameters are updated and they are updated explicitly to maximize ...
Lawnmower Man's user avatar
13 votes
Accepted

Are the dialogs at Sophia's (the robot) appearings scripted?

Sophia uses ChatScript. You can read about what ChatScript can do here. ChatScript keeps track of conversations with each user; can record where it is in a conversational flow and what facts it ...
Rebecca Merrett's user avatar
13 votes
Accepted

Why does ChatGPT create fake code?

ChatGPT is a generalised model. It does not understand any code. It does not know it is creating fake functions. It doesn't know that it isn't solving the question asked, because it doesn't know what ...
Rory Alsop's user avatar
12 votes

How can neural networks deal with varying input sizes?

In NLP you have an inherent ordering of the inputs so RNNs are a natural choice. For variable sized inputs where there is no particular ordering among the inputs, one can design networks which: use ...
Andre Holzner's user avatar
12 votes
Accepted

What is the purpose of Decoder mask (triangular mask) in Transformer?

The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens. So in order to predict next token, ...
Leo's user avatar
  • 300
11 votes
Accepted

Have any AI systems yet been developed that can knowingly lie to / deceive a human?

The Saturday Papers: Would AI Lie To You? is a blog post summarizing a research paper called Toward Characters Who Observe, Tell, Misremember, and Lie. This research paper details some researchers' ...
Left SE On 10_6_19's user avatar
11 votes
Accepted

Can an AI make a constructed (natural) language?

could an AI make/construct its own natural language, with words, conjugations, and grammar rules? Sure. This might be helpful: Simulated Evolution of Language: a Review of the Field Basically, a ...
Doxosophoi's user avatar
  • 1,945
11 votes
Accepted

Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

In statistics, if $X$ and $Y$ are independent and randomly distributed variables: $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\ Var(X + Y) = Var(X) + Var(Y) \\ \mathbb{E}[XY] = \mathbb{E}[X]\...
user3667125's user avatar
  • 1,570
11 votes

Why is ChatGPT bad at math?

(Check out my heavily related answer to a similar question here) Why is ChatGPT bad at math, while it is very good at other stuff? The problem comes down to the age-old problem of learning vs ...
Robin van Hoorn's user avatar
10 votes
Accepted

What are the specific requirements of the Turing test?

The "Turing Test" is generally taken to mean an updated version of the Imitation Game Alan Turing proposed in his 1951 paper of the same name. An early version had a human (male or female) and a ...
Rory Alsop's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible