Questions tagged [gpt]

For questions related to GPT (which stands for Generative Pre-Training), which is a combination of transformers (proposed in "Attention is All You Need") and unsupervised pre-training for solving language tasks, such as machine translation. GPT was proposed in "Improving Language Understanding by Generative Pre-Training" (2018) by Open AI. There's also GPT-2, which was proposed in "Language Models are Unsupervised Multitask Learners" (2019) by Open AI.

Filter by
Sorted by
Tagged with
1
vote
1answer
25 views

Is the Mask Needed for Masked Self-Attention During Inference with GPT-2

My understanding is that masked self-attention is necessary during training of GPT-2, as otherwise it would be able to directly see the correct next output at each iteration. My question is whether ...
0
votes
0answers
38 views

Training with many CPU cores doesn't improve performance

I ran my job on a computing cluster: first with 4 cores, then with 32 cores (2 nodes). But the training time is pretty much exactly the same for both of them: ~67 seconds per step. I am trying to ...
1
vote
0answers
14 views

How to use TPU for real-time low-latency inference?

I use Google's Cloud TPU hardware extensively using Tensorflow for training models and inference, however, when I run inference I do it in large batches. The TPU takes about 3 minutes to warm up ...
1
vote
1answer
49 views

Is it possible to use the GPT-2 model for time-series data prediction?

Is it possible and how trivial (or not) might it be (if possible) to retrain GPT-2 on time-series data instead of text?
1
vote
0answers
27 views

How to interpret a large variance of the loss function?

How do I interpret a large variance of a loss function? I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like ...
2
votes
0answers
34 views

How can I generate a document from a single word using GPT or BERT?

I have a dataset of 100000 documents each labelled with a topic to it. I want to create a model such that, given a topic, the model can generate a document from it. I came across language models GPT,...
4
votes
2answers
370 views

Where can I find pre-trained language models in English and German?

Where can I find (more) pre-trained language models? I am especially interested in neural network-based models for English and German. I am aware only of Language Model on One Billion Word Benchmark ...