35 votes
Accepted

Can BERT be used for sentence generating tasks?

For newbies, NO. Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its ...
soloice's user avatar
  • 511
30 votes

What is the intuition behind the dot product attention?

Let's start with a bit of notation and a couple of important clarifications. $\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word. $\...
Edoardo Guerriero's user avatar
29 votes
Accepted

How is BERT different from the original transformer architecture?

What is a transformer? The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-...
nbro's user avatar
  • 40.6k
16 votes

Why does GPT-2 Exclude the Transformer Encoder?

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
Faizy's user avatar
  • 1,114
9 votes

Can BERT be used for sentence generating tasks?

this experiment by Stephen Mayhew suggests that BERT is lousy at sequential text generation: http://mayhewsw.github.io/2019/01/16/can-bert-generate-text/ ...
stuart's user avatar
  • 191
7 votes

Isn't attention mask for BERT model useless?

This is just an implementation issue. One reason is the Huggingface implementation (which is not the original implementation by Google) wants to strictly separate the tokenization from the modeling. ...
Jindřich's user avatar
  • 391
6 votes

Why does GPT-2 Exclude the Transformer Encoder?

The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot ...
Ben's user avatar
  • 81
5 votes

What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?

Feedforward layer is an important part of the transformer architecture. Transformer architecture, in addition to the self-attention layer, that aggregates ...
spiridon_the_sun_rotator's user avatar
4 votes

What are the segment embeddings and position embeddings in BERT?

Sentences (for those tasks such as NLI which take two sentences as input) are differentiated in two ways in BERT: First, a [SEP] token is put between them Second, ...
finiteautomata's user avatar
4 votes
Accepted

How to use pre-trained BERT to extract the vectors from sentences?

Did you mean: How do you use a pre-trained BERT model in a feature-based setting to get pre-trained word contextual embeddings? Here is the BERT paper. I highly recommend you read it. Firstly, ...
samirzach's user avatar
  • 156
4 votes

Which positional encoding BERT use?

BERT uses trained position embeddings. The original paper does not say it explicitly, the term position embeddings (as opposed to encoding) suggests it is trained. When you look at BERT layers in ...
Jindřich's user avatar
  • 391
3 votes
Accepted

How do I calculate the probabilities of the BERT model prediction logits?

Your call to model.predict() is returning the logits for softmax. This is useful for training purposes. To get probabilties, you need to apply softmax on the logits....
Neil Slater's user avatar
  • 32.1k
3 votes

Adding BERT embeddings in LSTM embedding layer

Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead. ...
skillsmuggler's user avatar
3 votes
Accepted

Transformers: how to get the output (keys and values) of the encoder?

I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py). It seems like an extra linear layer learns ...
user3667125's user avatar
  • 1,570
3 votes
Accepted

What are the segment embeddings and position embeddings in BERT?

These embeddings are nothing more than token embeddings. You just randomly initialize them, then use gradient descent to train them, just like what you do with token embeddings.
soloice's user avatar
  • 511
3 votes
Accepted

Does BERT freeze the entire model body when it does fine-tuning?

Taken directly from HuggingFace Note that if you are used to freezing the body of your pretrained model (like in computer vision) the above may seem a bit strange, as we are directly fine-tuning the ...
Joon's user avatar
  • 51
3 votes
Accepted

What is the loss function and training task on which the original BERT model was trained

I found the following in the original BERT paper https://arxiv.org/pdf/1810.04805v2.pdf : "The training loss is the sum of the mean masked LM [language model] likelihood and the mean next ...
Ggjj11's user avatar
  • 188
3 votes

Why are some of the weights not initialized from the pretrained model checkpoint (from hugging face)?

This is because you are using a BertForSequenceClassification while the model you are using should be a BertForPreTraining. The difference is that BertForSequenceClassification has a classifier head ...
Lelouch's user avatar
  • 201
2 votes

Can BERT be used for sentence generating tasks?

No. Sentence generating is directly related to language modelling (given the previous words in the sentence, what is the next word). Because of bi-directionality of BERT, BERT cannot be used as a ...
Astariul's user avatar
  • 371
2 votes

Why does the BERT encoder have an intermediate layer between the attention and neural network layers with a bigger output?

The paper Undivided Attention: Are Intermediate Layers Necessary for BERT? should answer it. In the abstract, they write All BERT-based architectures have a self-attention block followed by a block ...
Tan Eugene's user avatar
2 votes
Accepted

Understanding how the loss was calculated for the SQuAD task in BERT paper

These answers are based on my personal understanding of Bert from both the paper and official_implementation, hope it will help: ...
HLeb's user avatar
  • 579
2 votes
Accepted

Why is my loss (binary cross entropy) converging on ~0.6? (Task: Natural Language Inference)

It seems to be overfitting and your model is not learning. Try SGD optimizer with a learning rate of 0.001 ADAM optimizer will give you a soon overfitting, and decreasing the learning rate will train ...
SahaTib's user avatar
  • 150
2 votes

How to use BERT as a multi-purpose conversational AI?

Think of BERT (or similar models) as as good starting place for understanding context. A couple options to make BERT contextualize dialogue: Concatenate all messages with a seperator embedding and ...
mshlis's user avatar
  • 2,359
2 votes

How to use speaker's information as well as text for fine-tuning BERT?

My answer assumes your fine-tuning architecture simply stacks a single fully-connected layer on top of the BERT [CLS] output, as in Figure 4b of the BERT paper. ...
primussucks's user avatar
2 votes
Accepted

How to keep track of the subject/entity in a sentence?

What you're describing is known as coreference resolution. More specifically, this example is anaphora resolution. The short answer is that this is an open research question and there is no well-...
CorruptedHeapScapeGoat's user avatar
2 votes
Accepted

Transformers: how does the decoder final layer output the desired token?

I found the answer by reading the paper referenced by that section, Using the output embedding to improve language models Based on this observation, we propose threeway weight tying (TWWT), where the ...
user3667125's user avatar
  • 1,570
2 votes

Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?

There is a pre-trained language model called ProphetNet for sequence-to-sequence learning with a novel self-supervised objective called future n-gram prediction. https://github.com/microsoft/...
usct01's user avatar
  • 121
2 votes
Accepted

fondamental question about regularization techniques to solve overfitting problem in neural networks

Regularization techniques reduce overfitting. This is why they tend to reduce training accuracy when applied to a model: they prevent the model to learn noise from the training data. For the same ...
Edoardo Guerriero's user avatar
2 votes
Accepted

How to classify data into organised groups by using a resulting classification vector and a set of probabilities?

To transform your logits [-5,1,2] into probabilities you can use the softmax function. This would get you ...
Snehal Patel's user avatar
2 votes
Accepted

Fine Tuning a Bert Transformer. How to label for emotions and train large scripts?

From my understanding of your task, you're looking to get the overall emotion classification score for a long piece of dialogue. BERT can handle contexts up to 512 tokens in length, so the task ...
Alexander Wan's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible