Questions tagged [gpt]

For questions related to GPT (which stands for Generative Pre-Training), which is a combination of transformers (proposed in "Attention is All You Need") and unsupervised pre-training for solving language tasks, such as machine translation. GPT was proposed in "Improving Language Understanding by Generative Pre-Training" (2018) by Open AI. There's also GPT-2, which was proposed in "Language Models are Unsupervised Multitask Learners" (2019) by Open AI.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

"Following instructions" as an emergent behaviour in transformer models - isn't this fundamentally different from the models' basic purpose?

I am not technically familiar with AI or neural networks beyond a tech news reading level of knowledge, so I apologise if this is a dumb question. I was recently reading this article on Ars Technica. ...
  • 101
2 votes
1 answer
37 views

Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?

Soft question here. I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million ...
0 votes
1 answer
113 views

Would it be possible to involve a proof assistant in the process of training a LLM?

LLMs like GPT-3 have been shown capable of outputting highly complex code. Sadly, actually using them to replace a programmer's job has two major caveats: LLMs are notoriously bad at producing ...
0 votes
0 answers
90 views

What subjects was ChatGPT trained on the most? Science/history/movies/reddit posts/wikipedia/books/news?

What subjects was ChatGPT trained on the most quantatively? It was trained on fiction and non-fiction books, wiki, and general web crawling. A bit of detective work tells me that compared to physics, ...
2 votes
1 answer
2k views

How do ChatGPT content filters work? (If not chatGPT then in general) [closed]

I first tried ChatGPT few days ago. And every day that goes by it seems more and more content filters are introduced. I can still make it do stuff if I "jailbreak it" but I feel like the ...
8 votes
3 answers
8k views

Why is ChatGPT bad at math?

As opposed to How does ChatGPT know math?, I've been seeing some things floating around the Twitterverse about how ChatGPT can actually be very bad at math. For instance, I asked it "If it takes ...
  • 2,355
0 votes
1 answer
149 views

Why don't OpenAI train a deep learning model to identify correct and incorrect information in ChatGPT's responses?

I'll preface this by saying that I have little experience in artificial intelligence, so this might be a naive question. However, in light of the recent controversy surrounding ChatGPT's inability to ...
  • 109
0 votes
1 answer
573 views

Fine-tune GPT-Neo with prompt and completion?

I'm new to AI and machine learning. To fine-tune GPT-3, I understand that we need a set of training examples that each consist of a single input ("prompt") and its associated output ("...
  • 101
0 votes
1 answer
22 views

can I add to a language model a prompt with output example?

I want to finetune GPT2 to extract relevant data from a given text. So for (a trivial) example, given the text "the car was manufactured in X, can reach Y km/h, and has Z horse powers", my ...
0 votes
0 answers
59 views

Left-to-Right vs Encoder-decoder Models

Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS) Left-to-Right: Auto-regressive, Left-to-right models, predict the ...
0 votes
0 answers
83 views

How do GPT models pass information for each token prediction?

So, I'm trying to understand what is going on in the following picture (from this paper): Each decoder blocker in the above GPT model has attention heads (red) and MLPs (green). I know that we add ...
0 votes
0 answers
47 views

Systematic analysis of the performance of BERT vs GPT for language generation?

BERT is generally not used for language generation, but it can be. The best comparison of performance between GPT-based and BERT-based generation I'm aware of is from is in the paper BERT has a Mouth ...
  • 380
0 votes
1 answer
745 views

How does GPT use the same embedding matrix for both input and output?

My understanding is that GPT uses the same embedding matrix for both inputs and output: Let $V$ be the vocab size, $D$ the number of embedding dimensions, and $E$ be a $V \times D$ embedding matrix: ...
1 vote
0 answers
49 views

How do I preload a conversational AI assistant like GPT-3 with complex relational data to draw on?

I'm exploring options to build a virtual assistant type of product. Creating good dialog is mostly solved with GPT-3 or even DialoGPT. My main question is how do I add larger amounts of relational ...
  • 111
0 votes
0 answers
73 views

Which index of the output in Transformers is used during inference to predict multiple words?

I am somewhat confused about how transformers, not just the original model, but also models like GPT-2 work when they are not training but are used multiple times to predict single tokens/words. The ...
1 vote
0 answers
187 views

Can you train GPT-J to use a specific list of words and prioritise them?

Can you train GPT-J to use a specific list of words and prioritise them? If so, please could you share how I would go about this? Say you're using GPT-J to write a story, you might wish to mention ...
0 votes
0 answers
72 views

Why is BERT/GPT capable of "for-all" generalization?

As shown in the figure: Why does token prediction work when "Socrates" is replaced with "Plato"? From the point of view of symbolic logic, the above example effectively performs ...
8 votes
1 answer
7k views

What is the "temperature" in the GPT models?

What does the temperature parameter mean when talking about the GPT models? I know that a higher temperature value means more randomness, but I want to know how randomness is introduced. Does ...
  • 203
0 votes
2 answers
3k views

How to fine-tune GPT-J with small dataset

I have followed this guide as closely as possible: https://github.com/kingoflolz/mesh-transformer-jax I'm trying to fine-tune GPT-J with a small dataset of ~500 lines: ...
2 votes
2 answers
214 views

Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

In music information retrieval, one usually converts an audio signal into some kind "sequence of frequency-vectors", such as STFT or Mel-spectrogram. I'm wondering if it is a good idea to ...
2 votes
0 answers
232 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
1 vote
1 answer
112 views

Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?

Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
  • 119
8 votes
3 answers
3k views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
2 votes
2 answers
1k views

Is it possible to integrate the GPT-3 by OpenAPI inside Unity3D or any game-engine?

My company has full access to beta testing for GPT-3. We wanted to try it for some games or game mechanics within Unity3D. Is it possible to use it for dialogues or with unity scripts? The Documents ...
6 votes
2 answers
922 views

What is the difference between the positional encoding techniques of the Transformer and GPT?

I know the original Transformer and the GPT (1-3) use two slightly different positional encoding techniques. More specifically, in GPT they say positional encoding is learned. What does that mean? ...
  • 275
1 vote
1 answer
163 views

Is GPT-3 an early example of strong AI in a narrow setting?

In GPT-2, the large achievement was being able to generate coherent text over a long-form while maintaining context. This was very impressive but for GPT-2 to do new language tasks, it had to be ...
  • 490
1 vote
0 answers
61 views

What is the efficiency of trained neural networks?

Training neural networks takes a while. My question is, how efficient is a neural network that is completely trained (assuming it's not a model that is constantly learning)? I understand that this is ...
  • 111
4 votes
0 answers
159 views

Can in principle GPT language models learn physics?

Does anyone know of research involving the GPT models to learn not only regular texts, but also learn from physics books with the equations written in latex format? My intuition is that the model ...
1 vote
2 answers
9k views

How much computing power does it cost to run GPT-3? [closed]

I know it cost around $4.3 million dollars to train, but how much computing power does it cost to run the finished program? IBM Watson chatbot AI only costs a few cents per chat message to use, ...
  • 121
1 vote
1 answer
478 views

How large should the corpus be to optimally retrain the GPT-2 model?

I just started working with the GPT-2 models and want to retrain one on a pretty narrow topic, so I have problems finding training material. How large should the corpus be to optimally retrain the GPT-...
16 votes
1 answer
7k views

What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?

When I studied neural networks, parameters were learning rate, batch size etc. But even GPT3's ArXiv paper does not mention anything about what exactly the parameters are, but gives a small hint that ...
  • 381
4 votes
1 answer
454 views

Why is GPT-3 such a game changer?

I've been hearing a lot about GPT-3 by OpenAI, and that it's a simple to use API with text in text out and has a big neural network off 175B parameters. But how did they achieve this huge number of ...
0 votes
1 answer
79 views

Is the size of a neural network directly linked with an increase in its inteligence?

Just came across this article on GPT-3, and that lead me to the question: In order to make a certain kind of neural network architecture smarter all one needs to do is to make it bigger? Also, if that ...
0 votes
1 answer
2k views

GPT-2: (Hardware) requirements for fine-tuning the 774M model [closed]

I wonder if there's anyone who has actually succeeded in fine-tuning GPT-2's 774M model without using cloud TPU's. My GeForce RTX 2070 SUPER couldn't handle it in previous attempts. I'm running ...
1 vote
0 answers
169 views

How can I use GPT-2 to modify seed text of one form into a different form (LENGTH INVARIANT) whilst retaining meaning?

I am currently starting a research project whereby I am trying to convert text of one form into another. i.e. If I were to write a seed sentance of the form "Scientists have finally achieved the ...
2 votes
0 answers
268 views

Pretrained Models for Keyword-Based Text Generation

I'm looking for an implementation that allows me to generate text based on a pre-trained model (e.g. GPT-2). An example would be gpt-2-keyword-generation (click here for demo). As the author notes, ...
2 votes
0 answers
265 views

Can we use GPT-2 to smooth out / correct text?

Are we able to use models like GPT-2 to smooth out/correct text? For instance if I have two paragraphs that need some text to make the transition easier to read, could this text be generated? And, ...
  • 121
3 votes
1 answer
885 views

Is the Mask Needed for Masked Self-Attention During Inference with GPT-2

My understanding is that masked self-attention is necessary during training of GPT-2, as otherwise it would be able to directly see the correct next output at each iteration. My question is whether ...
  • 31
1 vote
1 answer
918 views

Is it possible to use the GPT-2 model for time-series data prediction?

Is it possible and how trivial (or not) might it be (if possible) to retrain GPT-2 on time-series data instead of text?
  • 262
1 vote
0 answers
254 views

How to interpret a large variance of the loss function?

How do I interpret a large variance of a loss function? I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like ...
  • 260
2 votes
0 answers
64 views

How can I generate a document from a single word using GPT or BERT?

I have a dataset of 100000 documents each labelled with a topic to it. I want to create a model such that, given a topic, the model can generate a document from it. I came across language models GPT,...
7 votes
1 answer
657 views

How do we know if GPT-2 is a better language model?

You may have heard of GPT2, a new language model. It has recently attracted attention from the general public as the foundation that published the paper, OpenAI, ironically refused to share the whole ...
  • 232
5 votes
2 answers
2k views

Where can I find pre-trained language models in English and German? [closed]

Where can I find (more) pre-trained language models? I am especially interested in neural network-based models for English and German. I am aware only of Language Model on One Billion Word Benchmark ...