Skip to main content
41 votes
Accepted

What is the "temperature" in the GPT models?

In sequence generating models, for vocabulary of size $N$ (number of words, parts of words, any other kind of token), one predicts the next token from distribution of the form: $$ \mathrm{softmax} (...
spiridon_the_sun_rotator's user avatar
36 votes
Accepted

How does the (decoder-only) transformer architecture work?

Introduction Large-language models (LLMs) have gained tons of popularity lately with the releases of ChatGPT, GPT-4, Bard, and more. All these LLMs are based on the transformer neural network ...
Robin van Hoorn's user avatar
27 votes

Why is ChatGPT bad at math?

chatGPT is able to create well-formed sentences which contain phrases that are fitting for the input. It has rules extracted from its data, but those are not rules of understanding, but rules of '...
gctwnl's user avatar
  • 371
24 votes
Accepted

What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas ...
David's user avatar
  • 4,920
16 votes

Why does GPT-2 Exclude the Transformer Encoder?

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
Faizy's user avatar
  • 1,114
13 votes

How is GPT 4 able to solve math?

Large Language Models actually can do math. It's an "emergent" property, i.e. it appears only at larger scales. Understanding complex English language does require some analytical ability, ...
Harsh's user avatar
  • 1,315
11 votes

Why is ChatGPT bad at math?

(Check out my heavily related answer to a similar question here) Why is ChatGPT bad at math, while it is very good at other stuff? The problem comes down to the age-old problem of learning vs ...
Robin van Hoorn's user avatar
9 votes

Why don't OpenAI train a deep learning model to identify correct and incorrect information in ChatGPT's responses?

You are massively underestimating the difficulty of the task, you would need: A dataset containing labels of correct/incorrect, at a similar scale (billions of data points). A definition of correct/...
Dr. Snoopy's user avatar
  • 1,345
7 votes
Accepted

What sort of computer would be necessary to run queries on a LLM?

Executing specific queries, in the context of large language models, is referred to as inference. The hardware that runs GPT-4 has not been disclosed. However, Meta's LLaMA can be run on consumer ...
sjy's user avatar
  • 186
6 votes

Why does GPT-2 Exclude the Transformer Encoder?

The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot ...
Ben's user avatar
  • 81
6 votes
Accepted

What is the difference between the positional encoding techniques of the Transformer and GPT?

The purpose of introduction of positional encoding is to insert a notion of location of a given token in the sequence. Without it, due to the permutation equivariance (symmetry under the token ...
spiridon_the_sun_rotator's user avatar
6 votes

Is GPT-4 based on GPT-3 or was it trained from the scratch?

GPT-4 is largely based on GPT-3. From the GPT-4 Technical Report: GPT-4 is a Transformer-style model [39] The transformer-style model originates from the paper Attention Is All You Need, which lays ...
Minh-Long Luu's user avatar
5 votes

How is GPT 4 able to solve math?

ChatGPT now uses Wolfram Alpha to deal with math as well as other factual information. https://writings.stephenwolfram.com/2023/03/chatgpt-gets-its-wolfram-superpowers/
Jaume Oliver Lafont's user avatar
4 votes

Is GPT-3 an early example of strong AI in a narrow setting?

GPT-3 is based on in-context learning. It’s common wisdom one can hope that bigger models will yield better in-context capabilities. And indeed, this holds true, in the case of GPT-3 175B or "GPT-...
Arpit-Gole's user avatar
4 votes

Why is ChatGPT bad at math?

ChatGPT is good at math and can understand the logic. It can derive new conclusions on its own and generate value which was not there before. To make use of ChatGPT you have to provide it with the ...
Boris's user avatar
  • 49
4 votes
Accepted

Is GPT-4 based on GPT-3 or was it trained from the scratch?

GPT-4 is a transformer like GPT-3 and any other GPT. The training is certainly new, because it has a different size, you just can not transfer GPT-3 weights into an GPT-4 to continue training. The ...
Volker Siegel's user avatar
3 votes

Why is ChatGPT bad at math?

ChatGPT's GPT-4 model does not fall for this trap anymore due to more extensive training. I tested with two prompts: Prompt: If it takes 5 machines 5 minutes to make 5 devices, how long would it take ...
LeRobert's user avatar
3 votes
Accepted

Would it be possible to involve a proof assistant in the process of training a LLM?

Your post has links with: An article by Stephen Wolfram [ https://writings.stephenwolfram.com/2023/01/wolframalpha-as-the-way-to-bring-computational-knowledge-superpowers-to-chatgpt/ ], stating that ...
OlivierB's user avatar
3 votes
Accepted

Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two ...
Robz's user avatar
  • 204
3 votes
Accepted

Is it possible to integrate the GPT-3 by OpenAPI inside Unity3D or any game-engine?

Yes, OpenAI will release an API for GPT-3, so any developer can integrate it into their application. I don't believe the document for their API is public yet, so we don't know what the final interface ...
user3667125's user avatar
  • 1,590
3 votes

Is the Mask Needed for Masked Self-Attention During Inference with GPT-2

Answer to Q1) If sampling for next token do you need to apply mask during inference? Yes you do! The models ability to transfer information across positions was trained in this manner, and changing ...
mshlis's user avatar
  • 2,369
3 votes

Why is GPT-3 such a game changer?

The main point in GPT-3 and already in 2 was the observation that performance was steadily increasing with increasing model size (As seen in Figure 1.2 in your linked paper). So it seems that while ...
N. Kiefer's user avatar
  • 321
3 votes

What can GPT-4 do linguistics-wise?

So, here are some sentences GPT-4 generated in Russian (you can translate it with Google to English) and translated to PIE: Я люблю лес. Он полон жизни и красоты. H₁éǵʰōm h₂lewh₂óm. H₂ésti pl̥h₁nós ...
Anixx's user avatar
  • 351
3 votes
Accepted

How does transformer models like GPT generate valid meaningful response for meaningless garbage input?

If you give a human some input that doesn't seem to convey any meaning they will probably ask you for clarification. Presumably there are a lot of examples of this in the ChatGPT training data so that ...
quarague's user avatar
  • 291
3 votes

How is GPT 4 able to solve math?

As far as we know, GPT-4's core capabilities are still based mainly on a Large Language Model (LLM). If so, then the apparent capabilities to reason are a somewhat surprising emergent phenomenom from ...
Neil Slater's user avatar
  • 32.7k
3 votes

Process 2TB worth of conversational data hoarded over 40 years. How can I pass this into GPT to ask questions about it?

Two approaches that I am aware of: Chat your data This GitHub repository is accompanied by a blog post on how it works schematically. The overall approach is based on the LangChain library. Azure ...
Hans-Peter Schrei's user avatar
3 votes
Accepted

Aren't context lengths for transformers an artificial restriction?

Yes, you have the right idea. There's been a lot of work recently regarding extending the context-length of existing models, mostly looking at the Llama family of models. You should check out this ...
Alexander Wan's user avatar
3 votes

What is the current limit for LLMs regarding the size of the context that can be fed to them?

Most state of the art LLMs nowadays are based on an architecture called Transformer, which uses a technique called attention, which scales quadratically with the input size. Thus, for an input of 1 ...
Alberto's user avatar
  • 2,273
2 votes
Accepted

Is the size of a neural network directly linked with an increase in its inteligence?

First of all, there is no real 'intelligence' innate to artificial Neural Networks (NNs). All they do is trying to approximate a mathematical function with a certain degree of generalization (...
Daniel B.'s user avatar
  • 825
2 votes
Accepted

How much computing power does it cost to run GPT-3?

I can't anwser your question on how much computing power you might need, but you'll need atleast a smallgrid to run the biggest model just looking at the memory requirments (175B parameters so 700GB ...
hal9000's user avatar
  • 379

Only top scored, non community-wiki answers of a minimum length are eligible