34
votes
Accepted
Can BERT be used for sentence generating tasks?
For newbies, NO.
Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its ...
- 501
23
votes
Accepted
How is BERT different from the original transformer architecture?
What is a transformer?
The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-...
- 37.1k
22
votes
What is the intuition behind the dot product attention?
Let's start with a bit of notation and a couple of important clarifications.
$\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word.
$\...
- 4,773
8
votes
Why does GPT-2 Exclude the Transformer Encoder?
GPT-2 is a close copy of the basic transformer architecture.
GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
- 1,004
8
votes
Can BERT be used for sentence generating tasks?
this experiment by Stephen Mayhew suggests that BERT is lousy at sequential text generation:
http://mayhewsw.github.io/2019/01/16/can-bert-generate-text/
...
- 181
4
votes
What are the segment embeddings and position embeddings in BERT?
Sentences (for those tasks such as NLI which take two sentences as input) are differentiated in two ways in BERT:
First, a [SEP] token is put between them
Second, ...
- 141
4
votes
Accepted
How to use pre-trained BERT to extract the vectors from sentences?
Did you mean:
How do you use a pre-trained BERT model in a feature-based setting
to get pre-trained word contextual embeddings?
Here is the BERT paper. I highly recommend you read it.
Firstly, ...
- 156
4
votes
What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?
Feedforward layer is an important part of the transformer architecture.
Transformer architecture, in addition to the self-attention layer, that aggregates ...
- 2,194
3
votes
Why does GPT-2 Exclude the Transformer Encoder?
The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot ...
- 51
3
votes
Accepted
Transformers: how to get the output (keys and values) of the encoder?
I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py).
It seems like an extra linear layer learns ...
- 1,400
3
votes
Adding BERT embeddings in LSTM embedding layer
Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead.
...
- 393
3
votes
Accepted
What are the segment embeddings and position embeddings in BERT?
These embeddings are nothing more than token embeddings.
You just randomly initialize them, then use gradient descent to train them, just like what you do with token embeddings.
- 501
3
votes
Isn't attention mask for BERT model useless?
This is just an implementation issue. One reason is the Huggingface implementation (which is not the original implementation by Google) wants to strictly separate the tokenization from the modeling. ...
- 241
2
votes
Can BERT be used for sentence generating tasks?
No. Sentence generating is directly related to language modelling (given the previous words in the sentence, what is the next word). Because of bi-directionality of BERT, BERT cannot be used as a ...
- 371
2
votes
Accepted
Understanding how the loss was calculated for the SQuAD task in BERT paper
These answers are based on my personal understanding of Bert from both the paper and official_implementation, hope it will help:
...
- 499
2
votes
Accepted
Why is my loss (binary cross entropy) converging on ~0.6? (Task: Natural Language Inference)
It seems to be overfitting and your model is not learning. Try SGD optimizer with a learning rate of 0.001
ADAM optimizer will give you a soon overfitting, and decreasing the learning rate will train ...
- 140
2
votes
How to use BERT as a multi-purpose conversational AI?
Think of BERT (or similar models) as as good starting place for understanding context.
A couple options to make BERT contextualize dialogue:
Concatenate all messages with a seperator embedding and ...
- 2,339
2
votes
How to use speaker's information as well as text for fine-tuning BERT?
My answer assumes your fine-tuning architecture simply stacks a single fully-connected layer on top of the BERT [CLS] output, as in Figure 4b of the BERT paper.
...
- 346
2
votes
Accepted
Transformers: how does the decoder final layer output the desired token?
I found the answer by reading the paper referenced by that section, Using the output embedding to improve language models
Based on this observation, we propose threeway weight tying (TWWT), where the ...
- 1,400
2
votes
Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?
There is a pre-trained language model called ProphetNet for sequence-to-sequence learning with a novel self-supervised objective called future n-gram prediction.
https://github.com/microsoft/...
- 121
2
votes
Accepted
How do I calculate the probabilities of the BERT model prediction logits?
Your call to model.predict() is returning the logits for softmax. This is useful for training purposes.
To get probabilties, you need to apply softmax on the logits....
- 26.6k
2
votes
Accepted
How to keep track of the subject/entity in a sentence?
What you're describing is known as coreference resolution. More specifically, this example is anaphora resolution. The short answer is that this is an open research question and there is no well-...
2
votes
Accepted
Does BERT freeze the entire model body when it does fine-tuning?
Taken directly from HuggingFace
Note that if you are used to freezing the body of your pretrained model (like in computer vision) the above may seem a bit strange, as we are directly fine-tuning the ...
- 41
2
votes
Accepted
fondamental question about regularization techniques to solve overfitting problem in neural networks
Regularization techniques reduce overfitting. This is why they tend to reduce training accuracy when applied to a model: they prevent the model to learn noise from the training data. For the same ...
- 4,773
2
votes
Accepted
How to classify data into organised groups by using a resulting classification vector and a set of probabilities?
To transform your logits [-5,1,2] into probabilities you can use the softmax function. This would get you ...
- 882
1
vote
Should I need to use BERT embeddings while tokenizing using BERT tokenizer?
Generally speaking, the power of BERT for applications like NER is that the authors (of whichever implementation you use) performed a large-scale pretraining effort to create the embeddings. You can ...
- 266
1
vote
Accepted
1
vote
How to fine tune BERT for question answering?
The answer is yes but 'lightweight' will require a 'lightweight' model.
Your application for 'domain one' is called open domain question answering (ODQA). Here is a demonstration of ODQA using BERT: ...
- 1,698
1
vote
Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?
=> I don't have more ideas about BART and T5 Right now. but I had created a chatbot Based on GPT-2 Model-based on Microsoft DialoGPT. Which is fine-tuned on millions of parameters of Reddit. you can ...
- 355
1
vote
Accepted
Why does the BERT NSP head linear layer have two outputs?
This seems to be inherited from the original Google implementation, which also uses 2 outputs (https://github.com/google-research/bert/blob/master/run_pretraining.py#L293). I can see two possible ...
- 346
Only top scored, non community-wiki answers of a minimum length are eligible