Hot answers tagged

11 votes
Accepted

What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas ...
user avatar
8 votes
Accepted

What is the "temperature" in the GPT models?

In sequence generating models, for vocabulary of size $N$ (number of words, parts of words, any other kind of token), one predicts the next token from distribution of the form: $$ \mathrm{softmax} (...
user avatar
5 votes

Why does GPT-2 Exclude the Transformer Encoder?

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
user avatar
  • 627
3 votes
Accepted

Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two ...
user avatar
  • 194
3 votes
Accepted

Is it possible to integrate the GPT-3 by OpenAPI inside Unity3D or any game-engine?

Yes, OpenAI will release an API for GPT-3, so any developer can integrate it into their application. I don't believe the document for their API is public yet, so we don't know what the final interface ...
user avatar
  • 1,260
2 votes

Is the Mask Needed for Masked Self-Attention During Inference with GPT-2

Answer to Q1) If sampling for next token do you need to apply mask during inference? Yes you do! The models ability to transfer information across positions was trained in this manner, and changing ...
user avatar
  • 2,289
2 votes

Why is GPT-3 such a game changer?

The main point in GPT-3 and already in 2 was the observation that performance was steadily increasing with increasing model size (As seen in Figure 1.2 in your linked paper). So it seems that while ...
user avatar
  • 246
2 votes
Accepted

Is the size of a neural network directly linked with an increase in its inteligence?

First of all, there is no real 'intelligence' innate to artificial Neural Networks (NNs). All they do is trying to approximate a mathematical function with a certain degree of generalization (...
user avatar
  • 705
2 votes
Accepted

What is the difference between the positional encoding techniques of the Transformer and GPT?

The purpose of introduction of positional encoding is to insert a notion of location of a given token in the sequence. Without it, due to the permutation equivariance (symmetry under the token ...
user avatar
2 votes

Is GPT-3 an early example of strong AI in a narrow setting?

GPT-3 is based on in-context learning. It’s common wisdom one can hope that bigger models will yield better in-context capabilities. And indeed, this holds true, in the case of GPT-3 175B or "GPT-...
user avatar
2 votes
Accepted

How much computing power does it cost to run GPT-3?

I can't anwser your question on how much computing power you might need, but you'll need atleast a smallgrid to run the biggest model just looking at the memory requirments (175B parameters so 700GB ...
user avatar
  • 349
1 vote

Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

These papers are also very close to what I meant in the question (too long for a comment). The following references come mostly from work on speech recognition. Mockingjay In this work, they use an ...
user avatar
1 vote

Why does GPT-2 Exclude the Transformer Encoder?

The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot ...
user avatar
  • 21
1 vote

Is it possible to integrate the GPT-3 by OpenAPI inside Unity3D or any game-engine?

I found these links so hopefully they help. https://openai.com/blog/openai-api/ https://nordicapis.com/on-gpt-3-openai-and-apis/
user avatar
1 vote

How large should the corpus be to optimally retrain the GPT-2 model?

After doing many runs with GPT-2 355M I come to the conclusion that anything below 20k tokens yield worse and worse results the less tokens you have regardless of how many steps you train for. For my ...
user avatar
1 vote

How much computing power does it cost to run GPT-3?

I think it is premature to answer your question as OpenAI has not made GPT-3 available yet other than via a web-based API. For more information see OpenAI API. From OpenAI will start selling its text-...
user avatar
1 vote

GPT-2: (Hardware) requirements for fine-tuning the 774M model

Possibly a bit late to the answer, but I doubt you'd be able to run GPT-2 774M in FP32 on 2070 Super which has 8GB VRAM. I know it's not an exact comparison, but fine-tuning BERT Large (345M) in FP32 ...
user avatar
1 vote

Is it possible to use the GPT-2 model for time-series data prediction?

Definitely! but at that point it would be training a transformer-encoder (gpt2's architecture) and not GPT2 because GPT2 is defined by the weights / training procedure / data it was trained and not ...
user avatar
  • 2,289
1 vote

How do we know if GPT-2 is a better language model?

You have experimented with a small model (117M parameters). OpenAI has now released the medium sized model (345M parameters). Note that the full model has 1.5B parameters. Also note that GPT-2 has ...
user avatar
  • 121
1 vote
Accepted

Where can I find pre-trained language models in English and German?

Of course now there has been a huge development: Huggingface published pytorch-transformers, a library for the so successful Transformer models (BERT and its variants, GPT-2, XLNet, etc.), including ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible