# Tag Info

## Hot answers tagged gpt

Accepted

### What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas ...
• 4,190
Accepted

### What is the "temperature" in the GPT models?

In sequence generating models, for vocabulary of size $N$ (number of words, parts of words, any other kind of token), one predicts the next token from distribution of the form:  \mathrm{softmax} (...

### Why does GPT-2 Exclude the Transformer Encoder?

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
• 627
Accepted

### Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two ...
• 194
Accepted

### Is it possible to integrate the GPT-3 by OpenAPI inside Unity3D or any game-engine?

Yes, OpenAI will release an API for GPT-3, so any developer can integrate it into their application. I don't believe the document for their API is public yet, so we don't know what the final interface ...
• 1,260

### Is the Mask Needed for Masked Self-Attention During Inference with GPT-2

Answer to Q1) If sampling for next token do you need to apply mask during inference? Yes you do! The models ability to transfer information across positions was trained in this manner, and changing ...
• 2,289

### Why is GPT-3 such a game changer?

The main point in GPT-3 and already in 2 was the observation that performance was steadily increasing with increasing model size (As seen in Figure 1.2 in your linked paper). So it seems that while ...
• 246
Accepted

### Is the size of a neural network directly linked with an increase in its inteligence?

First of all, there is no real 'intelligence' innate to artificial Neural Networks (NNs). All they do is trying to approximate a mathematical function with a certain degree of generalization (...
• 705
Accepted

### What is the difference between the positional encoding techniques of the Transformer and GPT?

The purpose of introduction of positional encoding is to insert a notion of location of a given token in the sequence. Without it, due to the permutation equivariance (symmetry under the token ...

### Is GPT-3 an early example of strong AI in a narrow setting?

GPT-3 is based on in-context learning. It’s common wisdom one can hope that bigger models will yield better in-context capabilities. And indeed, this holds true, in the case of GPT-3 175B or "GPT-...
• 364
Accepted

### How much computing power does it cost to run GPT-3?

I can't anwser your question on how much computing power you might need, but you'll need atleast a smallgrid to run the biggest model just looking at the memory requirments (175B parameters so 700GB ...
• 349
1 vote

### Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

These papers are also very close to what I meant in the question (too long for a comment). The following references come mostly from work on speech recognition. Mockingjay In this work, they use an ...
1 vote

### Why does GPT-2 Exclude the Transformer Encoder?

The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot ...
• 21
1 vote

### Is it possible to integrate the GPT-3 by OpenAPI inside Unity3D or any game-engine?

I found these links so hopefully they help. https://openai.com/blog/openai-api/ https://nordicapis.com/on-gpt-3-openai-and-apis/
1 vote

### How large should the corpus be to optimally retrain the GPT-2 model?

After doing many runs with GPT-2 355M I come to the conclusion that anything below 20k tokens yield worse and worse results the less tokens you have regardless of how many steps you train for. For my ...
1 vote

### How much computing power does it cost to run GPT-3?

I think it is premature to answer your question as OpenAI has not made GPT-3 available yet other than via a web-based API. For more information see OpenAI API. From OpenAI will start selling its text-...
• 1,458
1 vote

### GPT-2: (Hardware) requirements for fine-tuning the 774M model

Possibly a bit late to the answer, but I doubt you'd be able to run GPT-2 774M in FP32 on 2070 Super which has 8GB VRAM. I know it's not an exact comparison, but fine-tuning BERT Large (345M) in FP32 ...
1 vote

### Is it possible to use the GPT-2 model for time-series data prediction?

Definitely! but at that point it would be training a transformer-encoder (gpt2's architecture) and not GPT2 because GPT2 is defined by the weights / training procedure / data it was trained and not ...
• 2,289
1 vote

### How do we know if GPT-2 is a better language model?

You have experimented with a small model (117M parameters). OpenAI has now released the medium sized model (345M parameters). Note that the full model has 1.5B parameters. Also note that GPT-2 has ...
• 121
1 vote
Accepted

### Where can I find pre-trained language models in English and German?

Of course now there has been a huge development: Huggingface published pytorch-transformers, a library for the so successful Transformer models (BERT and its variants, GPT-2, XLNet, etc.), including ...
• 161

Only top scored, non community-wiki answers of a minimum length are eligible