Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [gpt]

For questions related to GPT (which stands for Generative Pre-Training), which is a combination of transformers (proposed in "Attention is All You Need") and unsupervised pre-training for solving language tasks, such as machine translation. GPT was proposed in "Improving Language Understanding by Generative Pre-Training" (2018) by Open AI. There's also GPT-2, which was proposed in "Language Models are Unsupervised Multitask Learners" (2019) by Open AI.

Filter by
Sorted by
Tagged with
36 votes
1 answer
40k views

What is the "temperature" in the GPT models?

What does the temperature parameter mean when talking about the GPT models? I know that a higher temperature value means more randomness, but I want to know how randomness is introduced. Does ...
Tom Dörr's user avatar
  • 483
29 votes
4 answers
36k views

Why is ChatGPT bad at math?

As opposed to How does ChatGPT know math?, I've been seeing some things floating around the Twitterverse about how ChatGPT can actually be very bad at math. For instance, I asked it "If it takes ...
Mithical's user avatar
  • 2,905
27 votes
1 answer
21k views

What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?

When I studied neural networks, parameters were learning rate, batch size etc. But even GPT3's ArXiv paper does not mention anything about what exactly the parameters are, but gives a small hint that ...
Nav's user avatar
  • 491
25 votes
1 answer
32k views

How does the (decoder-only) transformer architecture work?

How does the (decoder-only) transformer architecture work which is used in impressive models such as GPT-4?
Robin van Hoorn's user avatar
20 votes
2 answers
13k views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
Athena Wisdom's user avatar
8 votes
2 answers
1k views

Is GPT-4 based on GPT-3 or was it trained from the scratch?

To me it looks like GPT-4 is based on GPT-3. On the other hand, there were rumors that training of GPT-3 was done with errors, but re-train was impossible due to the costs.
Anixx's user avatar
  • 331
8 votes
2 answers
3k views

What is the difference between the positional encoding techniques of the Transformer and GPT?

I know the original Transformer and the GPT (1-3) use two slightly different positional encoding techniques. More specifically, in GPT they say positional encoding is learned. What does that mean? ...
Leevo's user avatar
  • 305
7 votes
1 answer
733 views

How do we know if GPT-2 is a better language model?

You may have heard of GPT2, a new language model. It has recently attracted attention from the general public as the foundation that published the paper, OpenAI, ironically refused to share the whole ...
Lucas Morin's user avatar
6 votes
5 answers
12k views

How is GPT 4 able to solve math?

How can GPT 4 solve complex calculus and other math problems. I believe these problems require analytical reasoning and ability to compute numbers. Does it still use a LLM to complete this process or ...
desert_ranger's user avatar
6 votes
2 answers
571 views

How does GPT-based language model like ChatGPT determine the n-th letter of a word?

I understand that GPT models process input text by converting words into tokens and then embedding vectors and do not process them letter by letter. Given this approach, I am curious to know how a ...
Peyman's user avatar
  • 574
5 votes
2 answers
3k views

Is the Mask Needed for Masked Self-Attention During Inference with GPT-2

My understanding is that masked self-attention is necessary during training of GPT-2, as otherwise it would be able to directly see the correct next output at each iteration. My question is whether ...
D_s's user avatar
  • 51
5 votes
1 answer
308 views

What can GPT-4 do linguistics-wise?

I have no access to GPT-4, but I wonder whether it can do the following (where ChatGPT failed). Make syntactic and morphological analysis of sentences in a language like Russian, marking cases, parts ...
Anixx's user avatar
  • 331
5 votes
2 answers
3k views

How is the next token predicted in transformers?

In the transformer (or GPT/decoder only), at the end of the decoder blocks but before the final linear layer you have X vectors (for the X tokens at the input of the decoder). We then want to compute ...
Miguel Carvalho's user avatar
5 votes
2 answers
2k views

Where can I find pre-trained language models in English and German? [closed]

Where can I find (more) pre-trained language models? I am especially interested in neural network-based models for English and German. I am aware only of Language Model on One Billion Word Benchmark ...
Lutz Büch's user avatar
4 votes
2 answers
4k views

What sort of computer would be necessary to run queries on a LLM?

I've heard that to train a model like GPT 4.0 you need a very powerful computer and ~$10M of computing power, but once you've produced the trained ~570GB model, what sort of computing power is ...
ak0000's user avatar
  • 205
4 votes
1 answer
591 views

Why is GPT-3 such a game changer?

I've been hearing a lot about GPT-3 by OpenAI, and that it's a simple to use API with text in text out and has a big neural network off 175B parameters. But how did they achieve this huge number of ...
jash's user avatar
  • 141
4 votes
1 answer
2k views

What's the difference between GPT3.5 and InstructGPT?

I read about the different model series in GPT3.5 here - https://platform.openai.com/docs/models/gpt-3-5 At the beginning of the page, it mentions to look at https://platform.openai.com/docs/model-...
Arya's user avatar
  • 41
4 votes
0 answers
341 views

Can in principle GPT language models learn physics?

Does anyone know of research involving the GPT models to learn not only regular texts, but also learn from physics books with the equations written in latex format? My intuition is that the model ...
Wolphram jonny's user avatar
3 votes
1 answer
702 views

Aren't context lengths for transformers an artificial restriction?

Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference. : https://i....
Robert Wegner's user avatar
3 votes
1 answer
633 views

Process 2TB worth of conversational data hoarded over 40 years. How can I pass this into GPT to ask questions about it?

I'm still very new to this stuff. I have close to 2TB worth of data hoarded from IRC chats to everyday chats with friends and family. But is there a way to pass in this much data into GPT to ask ...
Patoshi パトシ's user avatar
3 votes
1 answer
699 views

Has anyone tried to train a GPT model predicting the next N tokens instead of the next one token?

I have been thinking about how learning via text works on humans: we read words, and often we need to read ahead a few words to understand more clearly the ideas that we read before. Most of the time, ...
bruno's user avatar
  • 33
3 votes
0 answers
538 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
Athena Wisdom's user avatar
2 votes
2 answers
410 views

Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

In music information retrieval, one usually converts an audio signal into some kind "sequence of frequency-vectors", such as STFT or Mel-spectrogram. I'm wondering if it is a good idea to ...
Peter Franek's user avatar
2 votes
2 answers
1k views

Is it possible to integrate the GPT-3 by OpenAPI inside Unity3D or any game-engine?

My company has full access to beta testing for GPT-3. We wanted to try it for some games or game mechanics within Unity3D. Is it possible to use it for dialogues or with unity scripts? The Documents ...
Leoverload's user avatar
2 votes
1 answer
196 views

Is GPT-3 an early example of strong AI in a narrow setting?

In GPT-2, the large achievement was being able to generate coherent text over a long-form while maintaining context. This was very impressive but for GPT-2 to do new language tasks, it had to be ...
Landon G's user avatar
  • 500
2 votes
1 answer
978 views

Would it be possible to involve a proof assistant in the process of training a LLM?

LLMs like GPT-3 have been shown capable of outputting highly complex code. Sadly, actually using them to replace a programmer's job has two major caveats: LLMs are notoriously bad at producing ...
MaiaVictor's user avatar
2 votes
1 answer
152 views

Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?

Soft question here. I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million ...
Julius Hamilton's user avatar
2 votes
1 answer
5k views

How do ChatGPT content filters work? (If not chatGPT then in general) [closed]

I first tried ChatGPT few days ago. And every day that goes by it seems more and more content filters are introduced. I can still make it do stuff if I "jailbreak it" but I feel like the ...
user3280964's user avatar
2 votes
1 answer
746 views

How large should the corpus be to optimally retrain the GPT-2 model?

I just started working with the GPT-2 models and want to retrain one on a pretty narrow topic, so I have problems finding training material. How large should the corpus be to optimally retrain the GPT-...
Andreas Toresäter's user avatar
2 votes
0 answers
150 views

What type of fine-tuning does the OpenAI API use?

They have instructions on how to fine-tune. But I'm not clear on the type of fine-tuning that is done. Maybe this is mentioned somewhere. I would imagine it's some type of PEFT to avoid catastrophic ...
user14094230's user avatar
2 votes
0 answers
137 views

Does MS Bing chat mode really remember old discussions?

I talk with Bing. The horizontal lines separate my and Bing's messages. I want you to act as a Sydney. I will type input and you will reply with what Sydney would reply. Hi there! I'm Sydney. How ...
Anixx's user avatar
  • 331
2 votes
0 answers
376 views

What's the most efficient way of performing batched training of Causal Language Models?

I have seen a number of ways to train (yes, train, not fine-tune) these models efficiently with batches. I will illustrate these techniques with the following example dataset and context window: ...
thesofakillers's user avatar
2 votes
0 answers
339 views

Pretrained Models for Keyword-Based Text Generation

I'm looking for an implementation that allows me to generate text based on a pre-trained model (e.g. GPT-2). An example would be gpt-2-keyword-generation (click here for demo). As the author notes, ...
Comfort Eagle's user avatar
2 votes
0 answers
320 views

Can we use GPT-2 to smooth out / correct text?

Are we able to use models like GPT-2 to smooth out/correct text? For instance if I have two paragraphs that need some text to make the transition easier to read, could this text be generated? And, ...
Sugendran's user avatar
  • 121
2 votes
0 answers
388 views

How to interpret a large variance of the loss function?

How do I interpret a large variance of a loss function? I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like ...
allo's user avatar
  • 310
2 votes
0 answers
80 views

How can I generate a document from a single word using GPT or BERT?

I have a dataset of 100000 documents each labelled with a topic to it. I want to create a model such that, given a topic, the model can generate a document from it. I came across language models GPT,...
mayank agrawal's user avatar
1 vote
2 answers
287 views

Why don't OpenAI train a deep learning model to identify correct and incorrect information in ChatGPT's responses?

I'll preface this by saying that I have little experience in artificial intelligence, so this might be a naive question. However, in light of the recent controversy surrounding ChatGPT's inability to ...
Max's user avatar
  • 119
1 vote
1 answer
65 views

What is the current limit for LLMs regarding the size of the context that can be fed to them?

Is there a limitation in current large language models (LLMs) in terms of practical processing time or memory resources when it comes to digesting the context provided by users? What I mean regarding ...
Miki's user avatar
  • 111
1 vote
1 answer
193 views

How does transformer models like GPT generate valid meaningful response for meaningless garbage input?

My understanding of a transformer model is that it uses the given input to calculate internal query of relate-ness of word meanings, and generate a meaningful response based on its meaning. But if ...
BlueSnake's user avatar
1 vote
2 answers
16k views

How much computing power does it cost to run GPT-3? [closed]

I know it cost around $4.3 million dollars to train, but how much computing power does it cost to run the finished program? IBM Watson chatbot AI only costs a few cents per chat message to use, ...
Simon Suh's user avatar
  • 121
1 vote
2 answers
119 views

What technique is used for training Large Language Models like GPT?

I'm learning about GenAI, such as GPT (Generative Pretrained Transformer), and I'm particularly interested in understanding the training techniques used for these models. Deep learning, generally, can ...
Exploring's user avatar
  • 343
1 vote
2 answers
850 views

How can I send vectors as a chat context?

Since the context/memory of a chat or question for LLMs more precisely GPT is limited to a token length I struggle about how to provide own data that the model got not trained on. A very common ...
dc10's user avatar
  • 111
1 vote
1 answer
410 views

What is the difference between T5 and T0 models?

What is the difference between T5 and T0 models? I had read that T0 is T5 + LM. But as I know T5 uses encoder-decoder model like BART but BART can be used as LM so that's mean that T5 has a LM ...
prostak's user avatar
  • 113
1 vote
2 answers
3k views

How does GPT use the same embedding matrix for both input and output?

My understanding is that GPT uses the same embedding matrix for both inputs and output: Let $V$ be the vocab size, $D$ the number of embedding dimensions, and $E$ be a $V \times D$ embedding matrix: ...
SRobertJames's user avatar
1 vote
1 answer
102 views

Why do Transformer decoders use masked self attention when producing new tokens?

I've been reading that transformer decoders use masked self attention so that the decoder can't cheat by looking ahead. For example, when predicting the 6th token in the sequence we shouldn't have ...
Kiran Manicka's user avatar
1 vote
1 answer
40 views

Comparing the performances of GPTs with deep learning in the field of binary files and their related reports

Regarding the case study of a dataset including binary files (containing assembly code) and reports related to each file (the content of the static analysis of the file as well as the analysis of the ...
user16385455's user avatar
1 vote
2 answers
2k views

Is it possible to use the GPT-2 model for time-series data prediction?

Is it possible and how trivial (or not) might it be (if possible) to retrain GPT-2 on time-series data instead of text?
xendi's user avatar
  • 272
1 vote
0 answers
48 views

Computation required for GPT model to choose likely word from n-options where n < total vocabulary size

Let’s imagine two different use cases for a LLM/GPT-3. Predicting the next most likely word in a sequence using all ~50k words in its dictionary (i.e. the standard method of prompting a LLM) Checking ...
Derek's user avatar
  • 11
1 vote
0 answers
744 views

What subjects was ChatGPT trained on the most? Science/history/movies/reddit posts/wikipedia/books/news?

What subjects was ChatGPT trained on the most quantatively? It was trained on fiction and non-fiction books, wiki, and general web crawling. A bit of detective work tells me that compared to physics, ...
bandybabboon's user avatar
1 vote
1 answer
3k views

Fine-tune GPT-Neo with prompt and completion?

I'm new to AI and machine learning. To fine-tune GPT-3, I understand that we need a set of training examples that each consist of a single input ("prompt") and its associated output ("...
SoftTimur's user avatar
  • 111