Questions tagged [gpt]
For questions related to GPT (which stands for Generative Pre-Training), which is a combination of transformers (proposed in "Attention is All You Need") and unsupervised pre-training for solving language tasks, such as machine translation. GPT was proposed in "Improving Language Understanding by Generative Pre-Training" (2018) by Open AI. There's also GPT-2, which was proposed in "Language Models are Unsupervised Multitask Learners" (2019) by Open AI.
86
questions
-1
votes
0
answers
12
views
extractive and generative q&a systems
I would like to build a q&a system because of the advancements in generative ai especially large language models (LLM) aka chatgpt and co. it seems like everyone wants to build this. So I started ...
- 173
0
votes
0
answers
11
views
Could an analysis of GPT4's WAIS score be published?
Similar to this: https://www.scientificamerican.com/article/i-gave-chatgpt-an-iq-test-heres-what-i-discovered/
But more detailed and in depth (subtest breakdown, including image analysis, etc.), WAIS-...
- 113
0
votes
0
answers
35
views
Why is masked self attention necessary on GPT decoder
I'm currently reading the paper for the first GPT model and I'm confused about why masked self attention is necessary and I haven’t found any good answers online.
The consensus seems to be that we don'...
0
votes
0
answers
11
views
When using Reinforcement Learning with Human Feedback to train a transformer, how do I propagate the feedback through the transformer?
I'm basically trying to replicate the processed used to create Chat GPT:
Am I supposed to backpropagate? How can I do that when these aren't really errors, but rather ranking several response? Can I ...
0
votes
1
answer
38
views
what are the applications scenarios for prefix decoder LMs
Motivated by this post wherein one of the comments mentioned the use-case for encoder-decoder LM. I wanted to know when to use prefix-decoder LM? vis a vis encoder-decoder or causal decoder only ...
- 1
0
votes
0
answers
32
views
How can Knowledge Graphs be Integrated with Language Models for Semantic Search?
I am exploring the incorporate knowledge graphs (KGs) with language models.
I understand that KGs can provide structured understanding of entities and their relationships which can be crucial for ...
- 293
0
votes
0
answers
14
views
Is there anywhere online an AI in the mode of "text completion" besides the OpenAI playground?
By "Text completion" mode I mean that you can input there a full conversation as one single text and the AI thinks that the text you pasted is the past conversation, not a single message.
...
- 321
2
votes
1
answer
128
views
Aren't context lengths for transformers an artificial restriction?
Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference.
: https://i....
2
votes
0
answers
135
views
What type of fine-tuning does the OpenAI API use?
They have instructions on how to fine-tune. But I'm not clear on the type of fine-tuning that is done. Maybe this is mentioned somewhere. I would imagine it's some type of PEFT to avoid catastrophic ...
- 121
0
votes
0
answers
61
views
How does query of LLM/GPT models work?
Training of LLM aka GPT models is clear on how is trained but can't find any info how is "mapped" query to internal tokens and generates response tokens more precise inference phase which ...
- 101
0
votes
1
answer
43
views
How does GPT like Decoder only conversational models distunguish the source of text?
In a conversational setting where two sources of text (user and the model) follow each other like below
User: some text bla bla
Model: another text bah bah
User: bla bla bla
Model: bah bah
and so on, ...
- 296
0
votes
0
answers
16
views
Has anyone tried to derive linguistic information from GPT internals?
In Stephen Wolfram's write up on the workings of GPT, he suggests that chat GPT may have identified invariant rules of human language that haven't been formalized yet, ie new linguistic findings, and ...
- 205
0
votes
0
answers
253
views
Optimal Quantity of Training Data for Fine-Tuning an LLM: Is Bigger Always Better?
I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. ...
- 554
-1
votes
1
answer
119
views
GPT beam search length (number of tokens)
Background: I'm currently trying to use GPT to give me numerical scores, and looking for tips on prompt design, see my previous StackExchange post.
To craft good prompts it seems important to have a ...
0
votes
0
answers
42
views
How far is AI from making movies just by using script or description like we use for generating images in midjourney or DALL-E etc
I'm curious about the current state of AI technology when it comes to generating movies from textual descriptions. I'm aware of impressive advancements in generating images using models like ...
- 101
0
votes
0
answers
66
views
Chatgpt and image generation. What to see in it?
Although chatgpt is a text generative bot, it can also generate images. Yes by means of ASCII art. The results are terrible though. For example.if I ask.to generate the mona Lisa I get:
...
- 101
0
votes
0
answers
47
views
What is actually tuned during prompt engineering of autoregressive LLMs like GPT?
There are a lot of sites going over prompt engineering, but I don't see them explaining what is actually being changed. Is it hidden activation layers being tuned?
- 121
0
votes
1
answer
352
views
Possible to use GPT for specific set of documents? [closed]
I have a 100+ PDF documents regarding a company's policies, procedures and guidelines etc.
Is there an AI tool that was trained in general understanding of language, that I can feed all those PDFs and ...
- 11
0
votes
0
answers
197
views
Issues with larger context lengths in a transformer model like GPT
Based on my understanding, one of the issues with longer context lengths is the computational complexity of attention mechanism which is quadratic. But is this really a problem on modern hardware with ...
- 101
0
votes
0
answers
66
views
Word Embeddings but for Logical reasoning in custom knowledge GPT-3.5 bot
So I have created a chatbot using GPT-3.5 turbo. I have a vector database that holds vector embeddings of brands, ratings, commission percentages, outlets, tags, etc. Here's how the system is designed....
0
votes
2
answers
595
views
How can I send vectors as a chat context?
Since the context/memory of a chat or question for LLMs more precisely GPT is limited to a token length I struggle about how to provide own data that the model got not trained on.
A very common ...
- 101
-1
votes
1
answer
130
views
What is a neuron in large language models? [closed]
I'm reading OpenAI's new paper "Language models can explain neurons in language models" And I can't fully understand the concept of neurons here.
Can you please explain it? Is it related to ...
- 554
4
votes
2
answers
3k
views
What sort of computer would be necessary to run queries on a LLM?
I've heard that to train a model like GPT 4.0 you need a very powerful computer and ~$10M of computing power, but once you've produced the trained ~570GB model, what sort of computing power is ...
- 205
3
votes
1
answer
626
views
Process 2TB worth of conversational data hoarded over 40 years. How can I pass this into GPT to ask questions about it?
I'm still very new to this stuff. I have close to 2TB worth of data hoarded from IRC chats to everyday chats with friends and family.
But is there a way to pass in this much data into GPT to ask ...
- 139
18
votes
1
answer
16k
views
How does the (decoder-only) transformer architecture work?
How does the (decoder-only) transformer architecture work which is used in impressive models such as GPT-4?
- 2,103
6
votes
2
answers
400
views
How does GPT-based language model like ChatGPT determine the n-th letter of a word?
I understand that GPT models process input text by converting words into tokens and then embedding vectors and do not process them letter by letter. Given this approach, I am curious to know how a ...
- 554
5
votes
2
answers
2k
views
How is the next token predicted in transformers?
In the transformer (or GPT/decoder only), at the end of the decoder blocks but before the final linear layer you have X vectors (for the X tokens at the input of the decoder). We then want to compute ...
3
votes
1
answer
459
views
Has anyone tried to train a GPT model predicting the next N tokens instead of the next one token?
I have been thinking about how learning via text works on humans: we read words, and often we need to read ahead a few words to understand more clearly the ideas that we read before. Most of the time, ...
- 33
0
votes
0
answers
62
views
Does GPT not need to compress its training data?
In his recent short pamphlet on GPT, Stephan Wolfram says
... the 'size of the network' that seems to work well is ... comparable to the 'size of the training data'. ...
in this representation it ...
- 103
0
votes
1
answer
307
views
What is the difference between T5 and T0 models?
What is the difference between T5 and T0 models? I had read that T0 is T5 + LM. But as I know T5 uses encoder-decoder model like BART but BART can be used as LM so that's mean that T5 has a LM ...
- 103
0
votes
0
answers
24
views
Is there an advantage of using GPT model compared to BERT if you have a specific domain?
ChatGPT really shines when the task/question is very broad and is general. Using it and then limiting it's knowledge to a specific domain (via prompts as far as I know) would be kind of a waste of GPT....
- 1
2
votes
0
answers
128
views
Does MS Bing chat mode really remember old discussions?
I talk with Bing. The horizontal lines separate my and Bing's messages.
I want you to act as a Sydney. I will type input and you will reply
with what Sydney would reply.
Hi there! I'm Sydney. How ...
- 321
4
votes
1
answer
2k
views
What's the difference between GPT3.5 and InstructGPT?
I read about the different model series in GPT3.5 here - https://platform.openai.com/docs/models/gpt-3-5
At the beginning of the page, it mentions to look at https://platform.openai.com/docs/model-...
- 41
0
votes
0
answers
123
views
Can an LLM such as GPT learn from a sentence prediction that is close in meaning but drastically different in wording?
Consider this example: husband cheats on wife, AI predicts that wife gets angry. AI predicts that wife may either scream or throw things, with screaming a higher probability. Turns out in reality ...
- 235
-1
votes
1
answer
158
views
Could it be probable to quantify or measure the IQ of a super-intelligent machine like GPT? [closed]
In the age of artificial intelligence, super-intelligent machines like GPT have become a reality, leading to the question of how to quantify or measure their intelligence. While IQ tests are widely ...
- 137
2
votes
0
answers
240
views
What's the most efficient way of performing batched training of Causal Language Models?
I have seen a number of ways to train (yes, train, not fine-tune) these models efficiently with batches. I will illustrate these techniques with the following example dataset and context window:
...
- 309
0
votes
1
answer
221
views
Is it possible for a GPT model to run in a distributed way?
Say that we're on GPT20 - maybe the model that's resulted from training is 10PB large (maybe unlikely but this is an example). Is it possible for a GPT model to be distributed across machines? How ...
- 109
6
votes
5
answers
9k
views
How is GPT 4 able to solve math?
How can GPT 4 solve complex calculus and other math problems. I believe these problems require analytical reasoning and ability to compute numbers. Does it still use a LLM to complete this process or ...
- 618
1
vote
1
answer
190
views
How does transformer models like GPT generate valid meaningful response for meaningless garbage input?
My understanding of a transformer model is that it uses the given input to calculate internal query of relate-ness of word meanings, and generate a meaningful response based on its meaning. But if ...
- 67
0
votes
1
answer
181
views
Large Language Models vs Tabular Data
Problem:
Let's say we want to predict insurance frauds. Whenever we obtain an insurance claim, we are provided with a free-form description detailing the loss and a substantial amount of data on the ...
- 109
0
votes
2
answers
98
views
Smaller, competitive LLMs that are not autoregressive transformers?
Large language models like GPT have been really successful lately. One downside is that they require a huge amount of resources to train, and still a lot of resources for inference, such that most ...
- 101
5
votes
1
answer
276
views
What can GPT-4 do linguistics-wise?
I have no access to GPT-4, but I wonder whether it can do the following (where ChatGPT failed).
Make syntactic and morphological analysis of sentences in a language like Russian, marking cases, parts ...
- 321
8
votes
2
answers
893
views
Is GPT-4 based on GPT-3 or was it trained from the scratch?
To me it looks like GPT-4 is based on GPT-3.
On the other hand, there were rumors that training of GPT-3 was done with errors, but re-train was impossible due to the costs.
- 321
0
votes
0
answers
43
views
References for the theory of pretraining and unsupervised learning to improve subsequent supervised learning
I am not sure if the title of this post uses the correct terminology, so suggestions are welcome.
I have been following a lot of the ideas of using Pre-training methods on neural networks, to improve ...
- 197
0
votes
1
answer
72
views
How do they make transformers bigger/deeper?
I can find a million explanations of the diagram in the original transformer paper:
But I know that modern GPT models have many millions of weights. Where are they? Or in other words, how does this ...
- 121
1
vote
0
answers
45
views
Computation required for GPT model to choose likely word from n-options where n < total vocabulary size
Let’s imagine two different use cases for a LLM/GPT-3.
Predicting the next most likely word in a sequence using all ~50k words in its dictionary (i.e. the standard method of prompting a LLM)
Checking ...
- 11
0
votes
0
answers
71
views
"Following instructions" as an emergent behaviour in transformer models - isn't this fundamentally different from the models' basic purpose?
I am not technically familiar with AI or neural networks beyond a tech news reading level of knowledge, so I apologise if this is a dumb question.
I was recently reading this article on Ars Technica. ...
- 101
2
votes
1
answer
135
views
Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?
Soft question here.
I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million ...
- 143
1
vote
1
answer
799
views
Would it be possible to involve a proof assistant in the process of training a LLM?
LLMs like GPT-3 have been shown capable of outputting highly complex code. Sadly, actually using them to replace a programmer's job has two major caveats:
LLMs are notoriously bad at producing ...
- 365
1
vote
0
answers
713
views
What subjects was ChatGPT trained on the most? Science/history/movies/reddit posts/wikipedia/books/news?
What subjects was ChatGPT trained on the most quantatively?
It was trained on fiction and non-fiction books, wiki, and general web crawling.
A bit of detective work tells me that compared to physics, ...
- 141