Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [gpt]

For questions related to GPT (which stands for Generative Pre-Training), which is a combination of transformers (proposed in "Attention is All You Need") and unsupervised pre-training for solving language tasks, such as machine translation. GPT was proposed in "Improving Language Understanding by Generative Pre-Training" (2018) by Open AI. There's also GPT-2, which was proposed in "Language Models are Unsupervised Multitask Learners" (2019) by Open AI.

Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

GPT Vision vs OCR & GPT text [closed]

I’m using GPT4 to create calendar events from a screenshot of a conversation. What’s better: Use GPT with vision and prompt Use OCR to pull out all text and then use a GPT text model
DD.'s user avatar
  • 101
0 votes
0 answers
34 views

Attention Mechanism: Why don't we just use a simple dot product instead of the Q, K, V matrices?

I am currently learning about Transformers by reading Richard Turner's paper "An Introduction to Transformers". On page 3 of the paper he gave a "naive" approach to build the ...
StockComCat's user avatar
0 votes
0 answers
16 views

Problems with understanding instruction fine-tuning

I'm trying to read up on instruction fine-tuning, but I think I have a big misunderstanding. As I understand, instruction datasets typically have 3 components: (a) the instruction (b) the output/...
Christian's user avatar
  • 101
0 votes
0 answers
29 views

Why doesn't my toy transformer model "grok"?

I'm working on reproducing the results by Neel Nanda on teaching a small transformer to perform modular addition: (operand_1+operand_2)%mod_value. The expectation ...
LawlessWalrus's user avatar
0 votes
0 answers
19 views

Are there leaderboards/tables/stats that compare inference times between close-sourced LLMs such as GPT 3.5/4 and Claude?

https://huggingface.co/spaces/optimum/llm-perf-leaderboard is great to compare inference times between LLMs but it misses close-sourced LLMs such as GPT 3.5/4 and Claude.
Franck Dernoncourt's user avatar
1 vote
2 answers
139 views

What technique is used for training Large Language Models like GPT?

I'm learning about GenAI, such as GPT (Generative Pretrained Transformer), and I'm particularly interested in understanding the training techniques used for these models. Deep learning, generally, can ...
Exploring's user avatar
  • 343
1 vote
1 answer
104 views

Why do Transformer decoders use masked self attention when producing new tokens?

I've been reading that transformer decoders use masked self attention so that the decoder can't cheat by looking ahead. For example, when predicting the 6th token in the sequence we shouldn't have ...
Kiran Manicka's user avatar
0 votes
1 answer
512 views

Getting started with training local LLM using python [closed]

As I'm completely new to this field, I find it hard to get started given the requirements I have. I'm a bit overwhelmed by all the models and options that are available. Even though it wasn't ...
Jeanluca Scaljeri's user avatar
0 votes
1 answer
75 views

Overcoming the quadratic scaling in transformer architecture

Do you know any papers that try to overcome quadratic scaling problems by attending lower dimensional representations in the dimension of tokens? For example, let's say that the input to the ...
Andy Yermakov's user avatar
1 vote
1 answer
40 views

Comparing the performances of GPTs with deep learning in the field of binary files and their related reports

Regarding the case study of a dataset including binary files (containing assembly code) and reports related to each file (the content of the static analysis of the file as well as the analysis of the ...
user16385455's user avatar
1 vote
1 answer
65 views

What is the current limit for LLMs regarding the size of the context that can be fed to them?

Is there a limitation in current large language models (LLMs) in terms of practical processing time or memory resources when it comes to digesting the context provided by users? What I mean regarding ...
Miki's user avatar
  • 111
0 votes
0 answers
18 views

Could an analysis of GPT4's WAIS score be published?

Similar to this: https://www.scientificamerican.com/article/i-gave-chatgpt-an-iq-test-heres-what-i-discovered/ But more detailed and in depth (subtest breakdown, including image analysis, etc.), WAIS-...
BigMistake's user avatar
0 votes
0 answers
43 views

When using Reinforcement Learning with Human Feedback to train a transformer, how do I propagate the feedback through the transformer?

I'm basically trying to replicate the processed used to create Chat GPT: Am I supposed to backpropagate? How can I do that when these aren't really errors, but rather ranking several response? Can I ...
Austin Capobianco's user avatar
0 votes
1 answer
239 views

what are the applications scenarios for prefix decoder LMs

Motivated by this post wherein one of the comments mentioned the use-case for encoder-decoder LM. I wanted to know when to use prefix-decoder LM? vis a vis encoder-decoder or causal decoder only ...
Singh's user avatar
  • 1
0 votes
0 answers
44 views

How can Knowledge Graphs be Integrated with Language Models for Semantic Search?

I am exploring the incorporate knowledge graphs (KGs) with language models. I understand that KGs can provide structured understanding of entities and their relationships which can be crucial for ...
Exploring's user avatar
  • 343
0 votes
0 answers
23 views

Is there anywhere online an AI in the mode of "text completion" besides the OpenAI playground?

By "Text completion" mode I mean that you can input there a full conversation as one single text and the AI thinks that the text you pasted is the past conversation, not a single message. ...
Anixx's user avatar
  • 331
3 votes
1 answer
744 views

Aren't context lengths for transformers an artificial restriction?

Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference. : https://i....
Robert Wegner's user avatar
2 votes
0 answers
151 views

What type of fine-tuning does the OpenAI API use?

They have instructions on how to fine-tune. But I'm not clear on the type of fine-tuning that is done. Maybe this is mentioned somewhere. I would imagine it's some type of PEFT to avoid catastrophic ...
user14094230's user avatar
0 votes
0 answers
77 views

How does query of LLM/GPT models work?

Training of LLM aka GPT models is clear on how is trained but can't find any info how is "mapped" query to internal tokens and generates response tokens more precise inference phase which ...
n1tk's user avatar
  • 101
0 votes
1 answer
58 views

How does GPT like Decoder only conversational models distunguish the source of text?

In a conversational setting where two sources of text (user and the model) follow each other like below User: some text bla bla Model: another text bah bah User: bla bla bla Model: bah bah and so on, ...
meliksahturker's user avatar
0 votes
0 answers
21 views

Has anyone tried to derive linguistic information from GPT internals?

In Stephen Wolfram's write up on the workings of GPT, he suggests that chat GPT may have identified invariant rules of human language that haven't been formalized yet, ie new linguistic findings, and ...
ak0000's user avatar
  • 205
0 votes
0 answers
334 views

Optimal Quantity of Training Data for Fine-Tuning an LLM: Is Bigger Always Better?

I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. ...
Peyman's user avatar
  • 574
-1 votes
1 answer
351 views

GPT beam search length (number of tokens)

Background: I'm currently trying to use GPT to give me numerical scores, and looking for tips on prompt design, see my previous StackExchange post. To craft good prompts it seems important to have a ...
just another mathmo's user avatar
0 votes
0 answers
45 views

How far is AI from making movies just by using script or description like we use for generating images in midjourney or DALL-E etc

I'm curious about the current state of AI technology when it comes to generating movies from textual descriptions. I'm aware of impressive advancements in generating images using models like ...
طلحة's user avatar
  • 101
0 votes
0 answers
70 views

Chatgpt and image generation. What to see in it?

Although chatgpt is a text generative bot, it can also generate images. Yes by means of ASCII art. The results are terrible though. For example.if I ask.to generate the mona Lisa I get: ...
gotch4's user avatar
  • 101
0 votes
0 answers
57 views

What is actually tuned during prompt engineering of autoregressive LLMs like GPT?

There are a lot of sites going over prompt engineering, but I don't see them explaining what is actually being changed. Is it hidden activation layers being tuned?
user14094230's user avatar
0 votes
1 answer
557 views

Possible to use GPT for specific set of documents? [closed]

I have a 100+ PDF documents regarding a company's policies, procedures and guidelines etc. Is there an AI tool that was trained in general understanding of language, that I can feed all those PDFs and ...
user9163823's user avatar
1 vote
2 answers
866 views

How can I send vectors as a chat context?

Since the context/memory of a chat or question for LLMs more precisely GPT is limited to a token length I struggle about how to provide own data that the model got not trained on. A very common ...
dc10's user avatar
  • 111
-1 votes
1 answer
134 views

What is a neuron in large language models? [closed]

I'm reading OpenAI's new paper "Language models can explain neurons in language models" And I can't fully understand the concept of neurons here. Can you please explain it? Is it related to ...
Peyman's user avatar
  • 574
4 votes
2 answers
4k views

What sort of computer would be necessary to run queries on a LLM?

I've heard that to train a model like GPT 4.0 you need a very powerful computer and ~$10M of computing power, but once you've produced the trained ~570GB model, what sort of computing power is ...
ak0000's user avatar
  • 205
3 votes
1 answer
633 views

Process 2TB worth of conversational data hoarded over 40 years. How can I pass this into GPT to ask questions about it?

I'm still very new to this stuff. I have close to 2TB worth of data hoarded from IRC chats to everyday chats with friends and family. But is there a way to pass in this much data into GPT to ask ...
Patoshi パトシ's user avatar
25 votes
1 answer
32k views

How does the (decoder-only) transformer architecture work?

How does the (decoder-only) transformer architecture work which is used in impressive models such as GPT-4?
Robin van Hoorn's user avatar
6 votes
2 answers
578 views

How does GPT-based language model like ChatGPT determine the n-th letter of a word?

I understand that GPT models process input text by converting words into tokens and then embedding vectors and do not process them letter by letter. Given this approach, I am curious to know how a ...
Peyman's user avatar
  • 574
5 votes
2 answers
3k views

How is the next token predicted in transformers?

In the transformer (or GPT/decoder only), at the end of the decoder blocks but before the final linear layer you have X vectors (for the X tokens at the input of the decoder). We then want to compute ...
Miguel Carvalho's user avatar
3 votes
1 answer
703 views

Has anyone tried to train a GPT model predicting the next N tokens instead of the next one token?

I have been thinking about how learning via text works on humans: we read words, and often we need to read ahead a few words to understand more clearly the ideas that we read before. Most of the time, ...
bruno's user avatar
  • 33
1 vote
1 answer
413 views

What is the difference between T5 and T0 models?

What is the difference between T5 and T0 models? I had read that T0 is T5 + LM. But as I know T5 uses encoder-decoder model like BART but BART can be used as LM so that's mean that T5 has a LM ...
prostak's user avatar
  • 113
2 votes
0 answers
137 views

Does MS Bing chat mode really remember old discussions?

I talk with Bing. The horizontal lines separate my and Bing's messages. I want you to act as a Sydney. I will type input and you will reply with what Sydney would reply. Hi there! I'm Sydney. How ...
Anixx's user avatar
  • 331
4 votes
1 answer
2k views

What's the difference between GPT3.5 and InstructGPT?

I read about the different model series in GPT3.5 here - https://platform.openai.com/docs/models/gpt-3-5 At the beginning of the page, it mentions to look at https://platform.openai.com/docs/model-...
Arya's user avatar
  • 41
-1 votes
1 answer
163 views

Could it be probable to quantify or measure the IQ of a super-intelligent machine like GPT? [closed]

In the age of artificial intelligence, super-intelligent machines like GPT have become a reality, leading to the question of how to quantify or measure their intelligence. While IQ tests are widely ...
R1-'s user avatar
  • 137
2 votes
0 answers
386 views

What's the most efficient way of performing batched training of Causal Language Models?

I have seen a number of ways to train (yes, train, not fine-tune) these models efficiently with batches. I will illustrate these techniques with the following example dataset and context window: ...
thesofakillers's user avatar
0 votes
1 answer
307 views

Is it possible for a GPT model to run in a distributed way?

Say that we're on GPT20 - maybe the model that's resulted from training is 10PB large (maybe unlikely but this is an example). Is it possible for a GPT model to be distributed across machines? How ...
alex's user avatar
  • 109
6 votes
5 answers
12k views

How is GPT 4 able to solve math?

How can GPT 4 solve complex calculus and other math problems. I believe these problems require analytical reasoning and ability to compute numbers. Does it still use a LLM to complete this process or ...
desert_ranger's user avatar
1 vote
1 answer
193 views

How does transformer models like GPT generate valid meaningful response for meaningless garbage input?

My understanding of a transformer model is that it uses the given input to calculate internal query of relate-ness of word meanings, and generate a meaningful response based on its meaning. But if ...
BlueSnake's user avatar
0 votes
1 answer
189 views

Large Language Models vs Tabular Data

Problem: Let's say we want to predict insurance frauds. Whenever we obtain an insurance claim, we are provided with a free-form description detailing the loss and a substantial amount of data on the ...
Glue's user avatar
  • 109
0 votes
2 answers
150 views

Smaller, competitive LLMs that are not autoregressive transformers?

Large language models like GPT have been really successful lately. One downside is that they require a huge amount of resources to train, and still a lot of resources for inference, such that most ...
jdm's user avatar
  • 101
5 votes
1 answer
308 views

What can GPT-4 do linguistics-wise?

I have no access to GPT-4, but I wonder whether it can do the following (where ChatGPT failed). Make syntactic and morphological analysis of sentences in a language like Russian, marking cases, parts ...
Anixx's user avatar
  • 331
8 votes
2 answers
1k views

Is GPT-4 based on GPT-3 or was it trained from the scratch?

To me it looks like GPT-4 is based on GPT-3. On the other hand, there were rumors that training of GPT-3 was done with errors, but re-train was impossible due to the costs.
Anixx's user avatar
  • 331
0 votes
0 answers
45 views

References for the theory of pretraining and unsupervised learning to improve subsequent supervised learning

I am not sure if the title of this post uses the correct terminology, so suggestions are welcome. I have been following a lot of the ideas of using Pre-training methods on neural networks, to improve ...
krishnab's user avatar
  • 207
0 votes
1 answer
85 views

How do they make transformers bigger/deeper?

I can find a million explanations of the diagram in the original transformer paper: But I know that modern GPT models have many millions of weights. Where are they? Or in other words, how does this ...
Mastiff's user avatar
  • 121
1 vote
0 answers
48 views

Computation required for GPT model to choose likely word from n-options where n < total vocabulary size

Let’s imagine two different use cases for a LLM/GPT-3. Predicting the next most likely word in a sequence using all ~50k words in its dictionary (i.e. the standard method of prompting a LLM) Checking ...
Derek's user avatar
  • 11