50 votes

Why does ChatGPT fail in playing "20 questions"?

Like any other question on why ChatGPT can't do something, the simple/superficial answer is that ChatGPT is just a language model fine-tuned with RL to be verbose and nice (or to answer like the human ...
nbro's user avatar
  • 40.5k
14 votes

Why does ChatGPT fail in playing "20 questions"?

It Wasn't Trained To A learning system performs best on the task for which it is given explicit feedback. That is the only time the parameters are updated and they are updated explicitly to maximize ...
Lawnmower Man's user avatar
10 votes

Why does ChatGPT fail in playing "20 questions"?

Because ChatGPT is not an artificial or synthetic intelligence, it's a large language model that possesses no intelligence in and of itself. It's able to simulate the appearance of intelligence by ...
Ian Kemp's user avatar
  • 205
10 votes

Do AI-based code-generators guarantee correct output?

They have no such guarantee. This is true for many reasons, but the simplest of which is: The definition of correct output is very underspecified. We do not have a rigorous way of defining exactly ...
chessprogrammer's user avatar
8 votes
Accepted

How are sentences numerically encoded before passing them to neural networks?

On a very basic level, you are absolutely correct about the encoding of the attached sentence. But, practically, when you have a set n number of documents to be encoded, things happen differently. ...
Chinmay's user avatar
  • 521
8 votes
Accepted

Why are biases (typically) not used in attention mechanism?

For certain types of layers, such as transformers and convolutional layers, including a bias term is unnecessary and adds unnecessary overhead to the model. The reason for this is that these layers ...
Marc Dumon's user avatar
7 votes
Accepted

Why different noise in GAN generate different images?

Let's take this apart. GANs stand for Generative Adversarial Networks. Your question is how GANs are generative (the G part of the name). For this we need to understand what they try to achieve. In ...
sfotiadis's user avatar
  • 301
5 votes

Does ChatGPT use different transformers for different downstream tasks?

It is just one huge model which performs autoregressive text generation. The ability to perform a wide variety of task, defined at inference time is called in-context learning and was introduced in ...
Ciodar's user avatar
  • 400
4 votes

What if we drop the causal mask in auto-regressive Transformer?

The purpose of the triangular causal mask in the attention mechanism is to enforce the autoregressive property of the model during training and inference. This property ensures that the model can only ...
Revolucion for Monica's user avatar
4 votes

Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

Assume that the query embeddings $Q$ and key embeddings $K$ have zero mean and unit std. Then for the variance of the attention score between any query and key we get: $$ \alpha = q_i k_j^T = \sum_{n=...
pi-tau's user avatar
  • 807
4 votes

Why different noise in GAN generate different images?

To further address you comment question, it's GAN's generator network's deep learning ability consisting of multiple layers of nonlinear transformations (e.g., convolutional layers, transposed ...
cinch's user avatar
  • 2,307
3 votes

Process 2TB worth of conversational data hoarded over 40 years. How can I pass this into GPT to ask questions about it?

Two approaches that I am aware of: Chat your data This GitHub repository is accompanied by a blog post on how it works schematically. The overall approach is based on the LangChain library. Azure ...
Hans-Peter Schrei's user avatar
3 votes

Why is ChatGPT bad at math?

ChatGPT's GPT-4 model does not fall for this trap anymore due to more extensive training. I tested with two prompts: Prompt: If it takes 5 machines 5 minutes to make 5 devices, how long would it take ...
LeRobert's user avatar
3 votes
Accepted

Aren't context lengths for transformers an artificial restriction?

Yes, you have the right idea. There's been a lot of work recently regarding extending the context-length of existing models, mostly looking at the Llama family of models. You should check out this ...
Alexander Wan's user avatar
3 votes

Are there strictly deterministic LLMs?

Any LLM that exists could easily be modified to be deterministic. At the present, they sample from a probability distribution for the next word. It is a trivial change to make, so that instead of ...
chessprogrammer's user avatar
3 votes

Is beam search the actual obstacle that prevents GPT-style models from doing sophisticated math reasoning?

I think that recently it has been a very interesting topic of conversation using LLMs for planning in the context of reasoning or for the case you mentioned for mathematical proving. Specially with ...
Cesar Ruiz's user avatar
3 votes

Do AI-based code-generators guarantee correct output?

Do AI based code generators guarantee correct output? No. There is no written guarantee by any provider. Researchers have measured "accuracy" (there's more than one way to think about this ...
Neil Slater's user avatar
  • 32.1k
2 votes
Accepted

How does GPT use the same embedding matrix for both input and output?

A GPT produces output based on its own previous output, so it must be able to understand its output. The learning input is provided as a stream of tokens, and these tokens are defined before learning ...
Volker Siegel's user avatar
2 votes

Why do the values in the cross attentional mechanism within a transformer come from the encoder and not from the decoder?

The idea of the cross-attention layer is to transform the input words to output words. The Decoder provides context of which input words should we pay attention to next based on the already decoded ...
Abhishek's user avatar
  • 121
2 votes

How do I create syntactically correct sentences given several words?

What a difference a few years make. When this was first asked in April 2019, the parts-of-speech tagging required for grammatical correctness was possible but hard, whilst the constraint of being ...
Neil Slater's user avatar
  • 32.1k
2 votes

How is the next token predicted in transformers?

Welcome to AI stack exchange! I understand the confusion. Inference (next token prediction) seems really counterintuitive and inefficient for transformers. And it is! The transformer is very efficient ...
Robin van Hoorn's user avatar
2 votes
Accepted

How does a Machine Learning model predict this classification problem?

when I don't give new sentence features (F1, F2) to my model and don't specify any procedure to calculate the features for new inputs, how the model can predict the sentiment of my new sentences? It ...
Luca Anzalone's user avatar
2 votes

If I freeze pre-trained model weights and than train a classifier on top of its embeddings does that called fine-tunning?

Wikipedia defines "fine-tuning" as: an approach to transfer learning in which the weights of a pre-trained model are trained on new data. Fine-tuning can be done on the entire neural ...
Kostya's user avatar
  • 2,524
2 votes

How do I choose a good treshold for classification (using cosine similarity scores)?

As correctly explained by @Robin van Hoorn, determining the classification threshold involves a trade-off between correct predictions and errors. One approach is to consider the TPR (true positive ...
Luca Anzalone's user avatar
2 votes

How do I choose a good treshold for classification (using cosine similarity scores)?

Determining the right classification/prediction threshold is always a trade-off between true positives, true negatives, false positives and false negatives. There is no universal guideline for ...
Robin van Hoorn's user avatar
2 votes

What information does the word embedding in Transformers will encode about the word when analysed outside of the model?

That’s a very interesting question. From what I understand, transformer embedding vectors are not as interpretable as word2vec embedding vectors regarding their meaning. They are more dependent on the ...
Chloe's user avatar
  • 21
2 votes
Accepted

Does fine-tuning a multilingual transformer model allow it to generalize to languages unseen in the fine-tuning dataset?

The short answer: Very unlikely. The extended answer: If you fine-tune a model, it becomes specialized for the type of data you fine-tune it on but you trade in some of its generalization capabilities....
emely_pi's user avatar
  • 277
2 votes
Accepted

How is the padding mask incorporated in the attention formula?

Entries of an attention mask are typically either $0$ or $-\infty$. So, adding such a mask gives either the original entry of $QK^T$ or $-\infty$. The issue with entrywise multiplication with a binary ...
Venna Banana's user avatar
2 votes

How does GPT like Decoder only conversational models distunguish the source of text?

A decoder-only conversational model, like GPT-3, generates text based on the context provided to it. It doesn't inherently "distinguish" the source of the text in the way humans might ...
DRV's user avatar
  • 1,683
2 votes
Accepted

Fine Tuning a Bert Transformer. How to label for emotions and train large scripts?

From my understanding of your task, you're looking to get the overall emotion classification score for a long piece of dialogue. BERT can handle contexts up to 512 tokens in length, so the task ...
Alexander Wan's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible