Questions tagged [gpt]

For questions related to GPT (which stands for Generative Pre-Training), which is a combination of transformers (proposed in "Attention is All You Need") and unsupervised pre-training for solving language tasks, such as machine translation. GPT was proposed in "Improving Language Understanding by Generative Pre-Training" (2018) by Open AI. There's also GPT-2, which was proposed in "Language Models are Unsupervised Multitask Learners" (2019) by Open AI.

Filter by
Sorted by
Tagged with
1 vote
1 answer
287 views

Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?

Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
6 votes
2 answers
545 views

How does GPT-based language model like ChatGPT determine the n-th letter of a word?

I understand that GPT models process input text by converting words into tokens and then embedding vectors and do not process them letter by letter. Given this approach, I am curious to know how a ...
0 votes
0 answers
25 views

Attention Mechanism: Why don't we just use a simple dot product instead of the Q, K, V matrices?

I am currently learning about Transformers by reading Richard Turner's paper "An Introduction to Transformers". On page 3 of the paper he gave a "naive" approach to build the ...
0 votes
0 answers
15 views

Problems with understanding instruction fine-tuning

I'm trying to read up on instruction fine-tuning, but I think I have a big misunderstanding. As I understand, instruction datasets typically have 3 components: (a) the instruction (b) the output/...
0 votes
0 answers
27 views

Why doesn't my toy transformer model "grok"?

I'm working on reproducing the results by Neel Nanda on teaching a small transformer to perform modular addition: (operand_1+operand_2)%mod_value. The expectation ...
0 votes
0 answers
19 views

Are there leaderboards/tables/stats that compare inference times between close-sourced LLMs such as GPT 3.5/4 and Claude?

https://huggingface.co/spaces/optimum/llm-perf-leaderboard is great to compare inference times between LLMs but it misses close-sourced LLMs such as GPT 3.5/4 and Claude.
1 vote
2 answers
88 views

What technique is used for training Large Language Models like GPT?

I'm learning about GenAI, such as GPT (Generative Pretrained Transformer), and I'm particularly interested in understanding the training techniques used for these models. Deep learning, generally, can ...
1 vote
1 answer
80 views

Why do Transformer decoders use masked self attention when producing new tokens?

I've been reading that transformer decoders use masked self attention so that the decoder can't cheat by looking ahead. For example, when predicting the 6th token in the sequence we shouldn't have ...
0 votes
2 answers
143 views

Smaller, competitive LLMs that are not autoregressive transformers?

Large language models like GPT have been really successful lately. One downside is that they require a huge amount of resources to train, and still a lot of resources for inference, such that most ...
0 votes
1 answer
338 views

Getting started with training local LLM using python [closed]

As I'm completely new to this field, I find it hard to get started given the requirements I have. I'm a bit overwhelmed by all the models and options that are available. Even though it wasn't ...
0 votes
2 answers
5k views

How to fine-tune GPT-J with small dataset

I have followed this guide as closely as possible: https://github.com/kingoflolz/mesh-transformer-jax I'm trying to fine-tune GPT-J with a small dataset of ~500 lines: ...
1 vote
1 answer
3k views

Fine-tune GPT-Neo with prompt and completion?

I'm new to AI and machine learning. To fine-tune GPT-3, I understand that we need a set of training examples that each consist of a single input ("prompt") and its associated output ("...
0 votes
1 answer
64 views

Overcoming the quadratic scaling in transformer architecture

Do you know any papers that try to overcome quadratic scaling problems by attending lower dimensional representations in the dimension of tokens? For example, let's say that the input to the ...
1 vote
1 answer
38 views

Comparing the performances of GPTs with deep learning in the field of binary files and their related reports

Regarding the case study of a dataset including binary files (containing assembly code) and reports related to each file (the content of the static analysis of the file as well as the analysis of the ...
1 vote
1 answer
65 views

What is the current limit for LLMs regarding the size of the context that can be fed to them?

Is there a limitation in current large language models (LLMs) in terms of practical processing time or memory resources when it comes to digesting the context provided by users? What I mean regarding ...
0 votes
0 answers
17 views

Could an analysis of GPT4's WAIS score be published?

Similar to this: https://www.scientificamerican.com/article/i-gave-chatgpt-an-iq-test-heres-what-i-discovered/ But more detailed and in depth (subtest breakdown, including image analysis, etc.), WAIS-...
0 votes
0 answers
44 views

How can Knowledge Graphs be Integrated with Language Models for Semantic Search?

I am exploring the incorporate knowledge graphs (KGs) with language models. I understand that KGs can provide structured understanding of entities and their relationships which can be crucial for ...
0 votes
0 answers
33 views

When using Reinforcement Learning with Human Feedback to train a transformer, how do I propagate the feedback through the transformer?

I'm basically trying to replicate the processed used to create Chat GPT: Am I supposed to backpropagate? How can I do that when these aren't really errors, but rather ranking several response? Can I ...
0 votes
1 answer
169 views

what are the applications scenarios for prefix decoder LMs

Motivated by this post wherein one of the comments mentioned the use-case for encoder-decoder LM. I wanted to know when to use prefix-decoder LM? vis a vis encoder-decoder or causal decoder only ...
5 votes
2 answers
2k views

Is the Mask Needed for Masked Self-Attention During Inference with GPT-2

My understanding is that masked self-attention is necessary during training of GPT-2, as otherwise it would be able to directly see the correct next output at each iteration. My question is whether ...
0 votes
0 answers
22 views

Is there anywhere online an AI in the mode of "text completion" besides the OpenAI playground?

By "Text completion" mode I mean that you can input there a full conversation as one single text and the AI thinks that the text you pasted is the past conversation, not a single message. ...
3 votes
1 answer
537 views

Aren't context lengths for transformers an artificial restriction?

Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference. : https://i....
2 votes
0 answers
149 views

What type of fine-tuning does the OpenAI API use?

They have instructions on how to fine-tune. But I'm not clear on the type of fine-tuning that is done. Maybe this is mentioned somewhere. I would imagine it's some type of PEFT to avoid catastrophic ...
1 vote
2 answers
759 views

How can I send vectors as a chat context?

Since the context/memory of a chat or question for LLMs more precisely GPT is limited to a token length I struggle about how to provide own data that the model got not trained on. A very common ...
0 votes
0 answers
72 views

How does query of LLM/GPT models work?

Training of LLM aka GPT models is clear on how is trained but can't find any info how is "mapped" query to internal tokens and generates response tokens more precise inference phase which ...
6 votes
5 answers
11k views

How is GPT 4 able to solve math?

How can GPT 4 solve complex calculus and other math problems. I believe these problems require analytical reasoning and ability to compute numbers. Does it still use a LLM to complete this process or ...
0 votes
1 answer
55 views

How does GPT like Decoder only conversational models distunguish the source of text?

In a conversational setting where two sources of text (user and the model) follow each other like below User: some text bla bla Model: another text bah bah User: bla bla bla Model: bah bah and so on, ...
0 votes
0 answers
21 views

Has anyone tried to derive linguistic information from GPT internals?

In Stephen Wolfram's write up on the workings of GPT, he suggests that chat GPT may have identified invariant rules of human language that haven't been formalized yet, ie new linguistic findings, and ...
0 votes
0 answers
333 views

Optimal Quantity of Training Data for Fine-Tuning an LLM: Is Bigger Always Better?

I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. ...
-1 votes
1 answer
291 views

GPT beam search length (number of tokens)

Background: I'm currently trying to use GPT to give me numerical scores, and looking for tips on prompt design, see my previous StackExchange post. To craft good prompts it seems important to have a ...
0 votes
0 answers
43 views

How far is AI from making movies just by using script or description like we use for generating images in midjourney or DALL-E etc

I'm curious about the current state of AI technology when it comes to generating movies from textual descriptions. I'm aware of impressive advancements in generating images using models like ...
1 vote
2 answers
286 views

Why don't OpenAI train a deep learning model to identify correct and incorrect information in ChatGPT's responses?

I'll preface this by saying that I have little experience in artificial intelligence, so this might be a naive question. However, in light of the recent controversy surrounding ChatGPT's inability to ...
5 votes
2 answers
3k views

How is the next token predicted in transformers?

In the transformer (or GPT/decoder only), at the end of the decoder blocks but before the final linear layer you have X vectors (for the X tokens at the input of the decoder). We then want to compute ...
0 votes
0 answers
69 views

Chatgpt and image generation. What to see in it?

Although chatgpt is a text generative bot, it can also generate images. Yes by means of ASCII art. The results are terrible though. For example.if I ask.to generate the mona Lisa I get: ...
0 votes
0 answers
56 views

What is actually tuned during prompt engineering of autoregressive LLMs like GPT?

There are a lot of sites going over prompt engineering, but I don't see them explaining what is actually being changed. Is it hidden activation layers being tuned?
23 votes
1 answer
29k views

How does the (decoder-only) transformer architecture work?

How does the (decoder-only) transformer architecture work which is used in impressive models such as GPT-4?
4 votes
2 answers
4k views

What sort of computer would be necessary to run queries on a LLM?

I've heard that to train a model like GPT 4.0 you need a very powerful computer and ~$10M of computing power, but once you've produced the trained ~570GB model, what sort of computing power is ...
0 votes
1 answer
479 views

Possible to use GPT for specific set of documents? [closed]

I have a 100+ PDF documents regarding a company's policies, procedures and guidelines etc. Is there an AI tool that was trained in general understanding of language, that I can feed all those PDFs and ...
29 votes
4 answers
36k views

Why is ChatGPT bad at math?

As opposed to How does ChatGPT know math?, I've been seeing some things floating around the Twitterverse about how ChatGPT can actually be very bad at math. For instance, I asked it "If it takes ...
0 votes
0 answers
222 views

Issues with larger context lengths in a transformer model like GPT

Based on my understanding, one of the issues with longer context lengths is the computational complexity of attention mechanism which is quadratic. But is this really a problem on modern hardware with ...
0 votes
0 answers
72 views

Word Embeddings but for Logical reasoning in custom knowledge GPT-3.5 bot

So I have created a chatbot using GPT-3.5 turbo. I have a vector database that holds vector embeddings of brands, ratings, commission percentages, outlets, tags, etc. Here's how the system is designed....
36 votes
1 answer
39k views

What is the "temperature" in the GPT models?

What does the temperature parameter mean when talking about the GPT models? I know that a higher temperature value means more randomness, but I want to know how randomness is introduced. Does ...
-1 votes
1 answer
133 views

What is a neuron in large language models? [closed]

I'm reading OpenAI's new paper "Language models can explain neurons in language models" And I can't fully understand the concept of neurons here. Can you please explain it? Is it related to ...
3 votes
1 answer
632 views

Process 2TB worth of conversational data hoarded over 40 years. How can I pass this into GPT to ask questions about it?

I'm still very new to this stuff. I have close to 2TB worth of data hoarded from IRC chats to everyday chats with friends and family. But is there a way to pass in this much data into GPT to ask ...
1 vote
2 answers
3k views

How does GPT use the same embedding matrix for both input and output?

My understanding is that GPT uses the same embedding matrix for both inputs and output: Let $V$ be the vocab size, $D$ the number of embedding dimensions, and $E$ be a $V \times D$ embedding matrix: ...
4 votes
1 answer
2k views

What's the difference between GPT3.5 and InstructGPT?

I read about the different model series in GPT3.5 here - https://platform.openai.com/docs/models/gpt-3-5 At the beginning of the page, it mentions to look at https://platform.openai.com/docs/model-...
3 votes
1 answer
664 views

Has anyone tried to train a GPT model predicting the next N tokens instead of the next one token?

I have been thinking about how learning via text works on humans: we read words, and often we need to read ahead a few words to understand more clearly the ideas that we read before. Most of the time, ...
-1 votes
1 answer
163 views

Could it be probable to quantify or measure the IQ of a super-intelligent machine like GPT? [closed]

In the age of artificial intelligence, super-intelligent machines like GPT have become a reality, leading to the question of how to quantify or measure their intelligence. While IQ tests are widely ...
0 votes
1 answer
189 views

Large Language Models vs Tabular Data

Problem: Let's say we want to predict insurance frauds. Whenever we obtain an insurance claim, we are provided with a free-form description detailing the loss and a substantial amount of data on the ...
0 votes
1 answer
387 views

What is the difference between T5 and T0 models?

What is the difference between T5 and T0 models? I had read that T0 is T5 + LM. But as I know T5 uses encoder-decoder model like BART but BART can be used as LM so that's mean that T5 has a LM ...