Questions tagged [gpt]

For questions related to GPT (which stands for Generative Pre-Training), which is a combination of transformers (proposed in "Attention is All You Need") and unsupervised pre-training for solving language tasks, such as machine translation. GPT was proposed in "Improving Language Understanding by Generative Pre-Training" (2018) by Open AI. There's also GPT-2, which was proposed in "Language Models are Unsupervised Multitask Learners" (2019) by Open AI.

39 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4 votes
0 answers
327 views

Can in principle GPT language models learn physics?

Does anyone know of research involving the GPT models to learn not only regular texts, but also learn from physics books with the equations written in latex format? My intuition is that the model ...
Wolphram jonny's user avatar
3 votes
0 answers
527 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
Athena Wisdom's user avatar
2 votes
0 answers
149 views

What type of fine-tuning does the OpenAI API use?

They have instructions on how to fine-tune. But I'm not clear on the type of fine-tuning that is done. Maybe this is mentioned somewhere. I would imagine it's some type of PEFT to avoid catastrophic ...
user14094230's user avatar
2 votes
0 answers
137 views

Does MS Bing chat mode really remember old discussions?

I talk with Bing. The horizontal lines separate my and Bing's messages. I want you to act as a Sydney. I will type input and you will reply with what Sydney would reply. Hi there! I'm Sydney. How ...
Anixx's user avatar
  • 331
2 votes
0 answers
359 views

What's the most efficient way of performing batched training of Causal Language Models?

I have seen a number of ways to train (yes, train, not fine-tune) these models efficiently with batches. I will illustrate these techniques with the following example dataset and context window: ...
thesofakillers's user avatar
2 votes
0 answers
339 views

Pretrained Models for Keyword-Based Text Generation

I'm looking for an implementation that allows me to generate text based on a pre-trained model (e.g. GPT-2). An example would be gpt-2-keyword-generation (click here for demo). As the author notes, ...
Comfort Eagle's user avatar
2 votes
0 answers
319 views

Can we use GPT-2 to smooth out / correct text?

Are we able to use models like GPT-2 to smooth out/correct text? For instance if I have two paragraphs that need some text to make the transition easier to read, could this text be generated? And, ...
Sugendran's user avatar
  • 121
2 votes
0 answers
383 views

How to interpret a large variance of the loss function?

How do I interpret a large variance of a loss function? I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like ...
allo's user avatar
  • 310
2 votes
0 answers
80 views

How can I generate a document from a single word using GPT or BERT?

I have a dataset of 100000 documents each labelled with a topic to it. I want to create a model such that, given a topic, the model can generate a document from it. I came across language models GPT,...
mayank agrawal's user avatar
1 vote
0 answers
48 views

Computation required for GPT model to choose likely word from n-options where n < total vocabulary size

Let’s imagine two different use cases for a LLM/GPT-3. Predicting the next most likely word in a sequence using all ~50k words in its dictionary (i.e. the standard method of prompting a LLM) Checking ...
Derek's user avatar
  • 11
1 vote
0 answers
738 views

What subjects was ChatGPT trained on the most? Science/history/movies/reddit posts/wikipedia/books/news?

What subjects was ChatGPT trained on the most quantatively? It was trained on fiction and non-fiction books, wiki, and general web crawling. A bit of detective work tells me that compared to physics, ...
bandybabboon's user avatar
1 vote
1 answer
3k views

Fine-tune GPT-Neo with prompt and completion?

I'm new to AI and machine learning. To fine-tune GPT-3, I understand that we need a set of training examples that each consist of a single input ("prompt") and its associated output ("...
SoftTimur's user avatar
  • 111
1 vote
0 answers
348 views

Left-to-Right vs Encoder-decoder Models

Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS) Left-to-Right: Auto-regressive, Left-to-right models, predict the ...
keyboardAnt's user avatar
1 vote
0 answers
194 views

How do I preload a conversational AI assistant like GPT-3 with complex relational data to draw on?

I'm exploring options to build a virtual assistant type of product. Creating good dialog is mostly solved with GPT-3 or even DialoGPT. My main question is how do I add larger amounts of relational ...
Sam7's user avatar
  • 111
1 vote
0 answers
200 views

Can you train GPT-J to use a specific list of words and prioritise them?

Can you train GPT-J to use a specific list of words and prioritise them? If so, please could you share how I would go about this? Say you're using GPT-J to write a story, you might wish to mention ...
Learningallday's user avatar
1 vote
1 answer
287 views

Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?

Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
Nyxynyx's user avatar
  • 119
1 vote
0 answers
85 views

What is the efficiency of trained neural networks?

Training neural networks takes a while. My question is, how efficient is a neural network that is completely trained (assuming it's not a model that is constantly learning)? I understand that this is ...
Anton's user avatar
  • 111
1 vote
0 answers
239 views

How can I use GPT-2 to modify seed text of one form into a different form (LENGTH INVARIANT) whilst retaining meaning?

I am currently starting a research project whereby I am trying to convert text of one form into another. i.e. If I were to write a seed sentance of the form "Scientists have finally achieved the ...
Colleen Larsen's user avatar
0 votes
0 answers
26 views

Attention Mechanism: Why don't we just use a simple dot product instead of the Q, K, V matrices?

I am currently learning about Transformers by reading Richard Turner's paper "An Introduction to Transformers". On page 3 of the paper he gave a "naive" approach to build the ...
StockComCat's user avatar
0 votes
0 answers
15 views

Problems with understanding instruction fine-tuning

I'm trying to read up on instruction fine-tuning, but I think I have a big misunderstanding. As I understand, instruction datasets typically have 3 components: (a) the instruction (b) the output/...
Christian's user avatar
  • 101
0 votes
0 answers
27 views

Why doesn't my toy transformer model "grok"?

I'm working on reproducing the results by Neel Nanda on teaching a small transformer to perform modular addition: (operand_1+operand_2)%mod_value. The expectation ...
LawlessWalrus's user avatar
0 votes
0 answers
19 views

Are there leaderboards/tables/stats that compare inference times between close-sourced LLMs such as GPT 3.5/4 and Claude?

https://huggingface.co/spaces/optimum/llm-perf-leaderboard is great to compare inference times between LLMs but it misses close-sourced LLMs such as GPT 3.5/4 and Claude.
Franck Dernoncourt's user avatar
0 votes
0 answers
17 views

Could an analysis of GPT4's WAIS score be published?

Similar to this: https://www.scientificamerican.com/article/i-gave-chatgpt-an-iq-test-heres-what-i-discovered/ But more detailed and in depth (subtest breakdown, including image analysis, etc.), WAIS-...
BigMistake's user avatar
0 votes
0 answers
38 views

When using Reinforcement Learning with Human Feedback to train a transformer, how do I propagate the feedback through the transformer?

I'm basically trying to replicate the processed used to create Chat GPT: Am I supposed to backpropagate? How can I do that when these aren't really errors, but rather ranking several response? Can I ...
Austin Capobianco's user avatar
0 votes
0 answers
44 views

How can Knowledge Graphs be Integrated with Language Models for Semantic Search?

I am exploring the incorporate knowledge graphs (KGs) with language models. I understand that KGs can provide structured understanding of entities and their relationships which can be crucial for ...
Exploring's user avatar
  • 343
0 votes
0 answers
22 views

Is there anywhere online an AI in the mode of "text completion" besides the OpenAI playground?

By "Text completion" mode I mean that you can input there a full conversation as one single text and the AI thinks that the text you pasted is the past conversation, not a single message. ...
Anixx's user avatar
  • 331
0 votes
0 answers
72 views

How does query of LLM/GPT models work?

Training of LLM aka GPT models is clear on how is trained but can't find any info how is "mapped" query to internal tokens and generates response tokens more precise inference phase which ...
n1tk's user avatar
  • 101
0 votes
0 answers
21 views

Has anyone tried to derive linguistic information from GPT internals?

In Stephen Wolfram's write up on the workings of GPT, he suggests that chat GPT may have identified invariant rules of human language that haven't been formalized yet, ie new linguistic findings, and ...
ak0000's user avatar
  • 205
0 votes
0 answers
333 views

Optimal Quantity of Training Data for Fine-Tuning an LLM: Is Bigger Always Better?

I am currently working on fine-tuning an LLM for a specific task, and I am trying to determine the optimal size for my training dataset. Intuitively, one might think that the more data, the better. ...
Peyman's user avatar
  • 564
0 votes
0 answers
43 views

How far is AI from making movies just by using script or description like we use for generating images in midjourney or DALL-E etc

I'm curious about the current state of AI technology when it comes to generating movies from textual descriptions. I'm aware of impressive advancements in generating images using models like ...
طلحة's user avatar
  • 101
0 votes
0 answers
69 views

Chatgpt and image generation. What to see in it?

Although chatgpt is a text generative bot, it can also generate images. Yes by means of ASCII art. The results are terrible though. For example.if I ask.to generate the mona Lisa I get: ...
gotch4's user avatar
  • 101
0 votes
0 answers
56 views

What is actually tuned during prompt engineering of autoregressive LLMs like GPT?

There are a lot of sites going over prompt engineering, but I don't see them explaining what is actually being changed. Is it hidden activation layers being tuned?
user14094230's user avatar
0 votes
0 answers
222 views

Issues with larger context lengths in a transformer model like GPT

Based on my understanding, one of the issues with longer context lengths is the computational complexity of attention mechanism which is quadratic. But is this really a problem on modern hardware with ...
rahul's user avatar
  • 101
0 votes
0 answers
73 views

Word Embeddings but for Logical reasoning in custom knowledge GPT-3.5 bot

So I have created a chatbot using GPT-3.5 turbo. I have a vector database that holds vector embeddings of brands, ratings, commission percentages, outlets, tags, etc. Here's how the system is designed....
Shahrukh Khan's user avatar
0 votes
2 answers
143 views

Smaller, competitive LLMs that are not autoregressive transformers?

Large language models like GPT have been really successful lately. One downside is that they require a huge amount of resources to train, and still a lot of resources for inference, such that most ...
jdm's user avatar
  • 101
0 votes
0 answers
44 views

References for the theory of pretraining and unsupervised learning to improve subsequent supervised learning

I am not sure if the title of this post uses the correct terminology, so suggestions are welcome. I have been following a lot of the ideas of using Pre-training methods on neural networks, to improve ...
krishnab's user avatar
  • 207
0 votes
0 answers
95 views

"Following instructions" as an emergent behaviour in transformer models - isn't this fundamentally different from the models' basic purpose?

I am not technically familiar with AI or neural networks beyond a tech news reading level of knowledge, so I apologise if this is a dumb question. I was recently reading this article on Ars Technica. ...
ShankarG's user avatar
  • 101
0 votes
0 answers
158 views

Why is BERT/GPT capable of "for-all" generalization?

As shown in the figure: Why does token prediction work when "Socrates" is replaced with "Plato"? From the point of view of symbolic logic, the above example effectively performs ...
Yan King Yin's user avatar
0 votes
2 answers
5k views

How to fine-tune GPT-J with small dataset

I have followed this guide as closely as possible: https://github.com/kingoflolz/mesh-transformer-jax I'm trying to fine-tune GPT-J with a small dataset of ~500 lines: ...
Ilya Karnaukhov's user avatar