59 votes

Was ChatGPT trained on Stack Overflow data?

ChatGPT is in the Large Language Models (LLM) category. The most (in)famous GPT model is probably GPT-3, because since then, researchers realized that LLMs mostly follow a predictable scaling law, ...
Minh-Long Luu's user avatar
41 votes
Accepted

What is the "temperature" in the GPT models?

In sequence generating models, for vocabulary of size $N$ (number of words, parts of words, any other kind of token), one predicts the next token from distribution of the form: $$ \mathrm{softmax} (...
spiridon_the_sun_rotator's user avatar
11 votes

How does ChatGPT know math?

Adding on to txopen's answer, it is interesting to note that for larger numbers with similar digits ChatGPT is unable to make any useful distinctions. For instance: Me: Which number is bigger: 1234.12 ...
Milo Moses's user avatar
9 votes

Why does ChatGPT not give the answer text all at once?

ChatGPT is a conversational-agent based on GPT3.5, which is a causal language model. Under the hood, GPT works by predicting the next token when provided with an input sequence of words. So yes, at ...
Rexcirus's user avatar
  • 1,174
8 votes
Accepted

What makes reproducing a model like GPT3/GPT3.5/ChatGPT difficult?

Challenges to reproduce ChatGPT: Compute cost Collect training data Find the proper choice for network architecture + RL (OpenAI hasn't published all the details) Two interesting papers on training ...
Franck Dernoncourt's user avatar
7 votes

What are pros and cons of Bi-LSTM as compared to LSTM?

I would say that the logic behind the introduction was more empirical than technical. The only difference between LSTM and Bi-LSTM is the possibility for Bi-LSTM to leverage future context chunks to ...
Edoardo Guerriero's user avatar
6 votes

How does ChatGPT know math?

I think that the dataset is so large and the model so well trained that it understood the probabilistic correlation of length in a token of numbers before a dot separation, and then the influence of ...
txopen's user avatar
  • 61
6 votes
Accepted

How was ChatGPT trained?

The key ingredient is called Reinforcement Learning from Human Feedback (RLHF), that is having humans rate the model answers and use the feedback to guide the model training. The official blog ...
Rexcirus's user avatar
  • 1,174
6 votes
Accepted

What is the difference between a language model and a word embedding?

Simplified: Word Embeddings does not consider context, Language Models does. For e.g Word2Vec, GloVe, or fastText, there exists one fixed vector per word. Think of the following two sentences: The ...
Isbister's user avatar
  • 186
5 votes

Why can't language models, like GPT-3, continuously learn once trained?

In theory, there is nothing stopping you from updating the weights of a neural network whenever you like. You run an example through the network, calculate the difference between the network's output ...
xojfqa's user avatar
  • 101
5 votes

How can a language model keep track of the provenance of the main knowledge/sources used to generate a given output?

Reddit user skyebreak pointed me to a relevant paper on that topic: Gao, Luyu, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao et al. "Attributed ...
Franck Dernoncourt's user avatar
4 votes
Accepted

What causes ChatGPT to generate responses that refer to itself as a bot or LM?

I don't work for openai, so I have no insight into exactly what works behind the scenes to make ChatGPT exhibit this behavior. However, in my opinion this is pretty clearly an example of prompt ...
RLC's user avatar
  • 214
4 votes

Why does ChatGPT not give the answer text all at once?

Why does ChatGPT not give the answer text all at once? Because ChatGPT is autoregressive (=generates each new word by looking at previous words), as Rexcirus mentioned. Is this just for show? On ...
Franck Dernoncourt's user avatar
4 votes

Is there a limitation to the amount of data that a genAI model could be trained upon?

I'll try to deconstruct your question and give you the most informative answer: Is there a limitation to the amount of data that a genAI model could be trained upon? In the way that this question is ...
Robin van Hoorn's user avatar
3 votes

Why do language models produce different outputs for same prompt?

Language models produce a probability distribution over a set of words. You determine the next word by sampling from this distribution. So, determining the next word is stochastic even though the ...
SpiderRico's user avatar
  • 1,000
3 votes

What is the difference between a language model and a word embedding?

A language model aims to estimate the probability of one or more words given the surrounding words. Given a sentence composed of $w_{1},...,w_{i-1},\_ , w_{i+1},..,w_{n}$, you can find which is the i-...
SMattia's user avatar
  • 41
3 votes

What are the main differences between a language model and a machine translation model?

The simple language model will give you the probability of a sequence of tokens(sentence) for that language. So lets say if you have trained a model for English language your model can give you the ...
Jayendra Parmar's user avatar
3 votes
Accepted

Are there any good alternatives to an LSTM language model for text generation?

The current state of the art in natural language generation are all auto-regressive transformer models. Transformers no longer use recurrent neural networks such as LSTM, because the recurrences makes ...
user3667125's user avatar
  • 1,570
3 votes
Accepted

What background should I have before starting to fine tune a Large Language Model?

There are considerable free and excellent resources out there (in the wild). You can check The Stanford Natural Language Processing Group teaching page; you can easily follow their YouTube courses on ...
Eduard's user avatar
  • 211
3 votes

What puts the "chat" in a system like ChatGPT?

TLDR: It is taught to do that during training The secret sauce here is not in the architecture but in the fine-tuning part of the training process. The details of which have not been made public yet. ...
Robin van Hoorn's user avatar
3 votes

Is beam search the actual obstacle that prevents GPT-style models from doing sophisticated math reasoning?

I think that recently it has been a very interesting topic of conversation using LLMs for planning in the context of reasoning or for the case you mentioned for mathematical proving. Specially with ...
Cesar Ruiz's user avatar
2 votes

What would be a good internal language for an AI?

I think the first question you should answer is: "What questions should the AI be able to answer?" If the intend was that the AI should be able to answer any questions, then that is simply not doable (...
Henriette Harmse's user avatar
2 votes

What would be a good internal language for an AI?

This is (even though it doesn't look like it at first glance) a deeply philosophical question about the nature of 'meaning'. This answer is necessarily limited in scope. There are many ways of ...
Oliver Mason's user avatar
  • 5,387
2 votes

Are embeddings in multi-lingual language models comparable across languages?

Embeddings generated by transformers like Bert or XLM-R are fundamentally different from embeddings learned through language models like GloVe or Word2Vec. The latter are static, i.e. they are just ...
Edoardo Guerriero's user avatar
2 votes

How to use BERT as a multi-purpose conversational AI?

Think of BERT (or similar models) as as good starting place for understanding context. A couple options to make BERT contextualize dialogue: Concatenate all messages with a seperator embedding and ...
mshlis's user avatar
  • 2,359
2 votes

Was ChatGPT trained on Stack Overflow data?

Technically speaking, we don't know whether ChatGPT used Stack Overflow data for the training. That is because ChatGPT is a proprietary model, and OpenAI hasn't published training details to the ...
pcpthm's user avatar
  • 266
2 votes
Accepted

For specific tasks, is it better to fine-tune models on examples or just use prompting with the context of the task?

Fine tuning is superior, since the whole network specialise to solve a given problem only. A specialist will always beat a generalist in the specialised task. That said, if the generalist network is ...
Rexcirus's user avatar
  • 1,174
2 votes

What makes reproducing a model like GPT3/GPT3.5/ChatGPT difficult?

Actually, Google created a bigger model than GPT-3 and models in the GPT-3.5 series, and consequently ChatGPT too (because ChatGPT is based on a GPT-3.5 model) - Switch-C has trillions of parameters, ...
nbro's user avatar
  • 40.5k
2 votes
Accepted

Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?

Yes, with caveats. Yes: If the data covers a niche and is very rare, you can indeed fine-tune a large model to your needs. Caveats: Fine tuning a model still require significant compute. Moreover the ...
Rexcirus's user avatar
  • 1,174
2 votes
Accepted

How do temperature and repetition penalty interfere?

TL;DR: Temperature is applied after repetition penalty, so it smoothes out its effect. They are basically independent hyper-parameters of the decoding, but applied after each other. Higher temperature ...
Jindřich's user avatar
  • 391

Only top scored, non community-wiki answers of a minimum length are eligible