Hot answers tagged

30 votes
Accepted

Can BERT be used for sentence generating tasks?

For newbies, NO. Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its ...
user avatar
  • 461
18 votes

What is the intuition behind the dot product attention?

Let's start with a bit of notation and a couple of important clarifications. $\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word. $\...
user avatar
16 votes
Accepted

How is BERT different from the original transformer architecture?

What is a transformer? The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-...
user avatar
  • 34.4k
5 votes

Why does GPT-2 Exclude the Transformer Encoder?

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
user avatar
  • 627
5 votes

Can BERT be used for sentence generating tasks?

this experiment by Stephen Mayhew suggests that BERT is lousy at sequential text generation: http://mayhewsw.github.io/2019/01/16/can-bert-generate-text/ ...
user avatar
  • 151
4 votes

What are the segment embeddings and position embeddings in BERT?

Sentences (for those tasks such as NLI which take two sentences as input) are differentiated in two ways in BERT: First, a [SEP] token is put between them Second, ...
user avatar
3 votes
Accepted

Transformers: how to get the output (keys and values) of the encoder?

I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py). It seems like an extra linear layer learns ...
user avatar
  • 1,260
3 votes
Accepted

How to use pre-trained BERT to extract the vectors from sentences?

Did you mean: How do you use a pre-trained BERT model in a feature-based setting to get pre-trained word contextual embeddings? Here is the BERT paper. I highly recommend you read it. Firstly, ...
user avatar
  • 146
3 votes

Adding BERT embeddings in LSTM embedding layer

Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead. ...
user avatar
3 votes
Accepted

What are the segment embeddings and position embeddings in BERT?

These embeddings are nothing more than token embeddings. You just randomly initialize them, then use gradient descent to train them, just like what you do with token embeddings.
user avatar
  • 461
3 votes

Isn't attention mask for BERT model useless?

This is just an implementation issue. One reason is the Huggingface implementation (which is not the original implementation by Google) wants to strictly separate the tokenization from the modeling. ...
user avatar
  • 141
3 votes

What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?

Feedforward layer is an important part of the transformer architecture. Transformer architecture, in addition to the self-attention layer, that aggregates ...
user avatar
2 votes

Can BERT be used for sentence generating tasks?

No. Sentence generating is directly related to language modelling (given the previous words in the sentence, what is the next word). Because of bi-directionality of BERT, BERT cannot be used as a ...
user avatar
  • 361
2 votes

How to use BERT as a multi-purpose conversational AI?

Think of BERT (or similar models) as as good starting place for understanding context. A couple options to make BERT contextualize dialogue: Concatenate all messages with a seperator embedding and ...
user avatar
  • 2,289
2 votes

How to use speaker's information as well as text for fine-tuning BERT?

My answer assumes your fine-tuning architecture simply stacks a single fully-connected layer on top of the BERT [CLS] output, as in Figure 4b of the BERT paper. ...
user avatar
2 votes
Accepted

Transformers: how does the decoder final layer output the desired token?

I found the answer by reading the paper referenced by that section, Using the output embedding to improve language models Based on this observation, we propose threeway weight tying (TWWT), where the ...
user avatar
  • 1,260
2 votes

Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?

There is a pre-trained language model called ProphetNet for sequence-to-sequence learning with a novel self-supervised objective called future n-gram prediction. https://github.com/microsoft/...
user avatar
  • 121
2 votes
Accepted

How to keep track of the subject/entity in a sentence?

What you're describing is known as coreference resolution. More specifically, this example is anaphora resolution. The short answer is that this is an open research question and there is no well-...
user avatar
2 votes
Accepted

fondamental question about regularization techniques to solve overfitting problem in neural networks

Regularization techniques reduce overfitting. This is why they tend to reduce training accuracy when applied to a model: they prevent the model to learn noise from the training data. For the same ...
user avatar
1 vote
Accepted

Does BERT freeze the entire model body when it does fine-tuning?

Taken directly from HuggingFace Note that if you are used to freezing the body of your pretrained model (like in computer vision) the above may seem a bit strange, as we are directly fine-tuning the ...
user avatar
  • 21
1 vote

Why does GPT-2 Exclude the Transformer Encoder?

The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot ...
user avatar
  • 21
1 vote

Should I need to use BERT embeddings while tokenizing using BERT tokenizer?

Generally speaking, the power of BERT for applications like NER is that the authors (of whichever implementation you use) performed a large-scale pretraining effort to create the embeddings. You can ...
user avatar
  • 266
1 vote
Accepted

How do I calculate the probabilities of the BERT model prediction logits?

Your call to model.predict() is returning the logits for softmax. This is useful for training purposes. To get probabilties, you need to apply softmax on the logits....
user avatar
  • 24.5k
1 vote
Accepted

Sentiment analysis does not handle neturals

Yes, there is. You can try Spacy. Here you go. ...
user avatar
1 vote

How to fine tune BERT for question answering?

The answer is yes but 'lightweight' will require a 'lightweight' model. Your application for 'domain one' is called open domain question answering (ODQA). Here is a demonstration of ODQA using BERT: ...
user avatar
1 vote
Accepted

Why does the BERT NSP head linear layer have two outputs?

This seems to be inherited from the original Google implementation, which also uses 2 outputs (https://github.com/google-research/bert/blob/master/run_pretraining.py#L293). I can see two possible ...
user avatar
1 vote

How to add a pretrained model to my layers to get embeddings?

I think you should use Keras embedding layer. It will be too easier than what you are doing. Steps Create Embedding Matrix add matrix to embedding layer while building model. You will find detailed ...
user avatar
  • 111
1 vote
Accepted

Why is my loss (binary cross entropy) converging on ~0.6? (Task: Natural Language Inference)

It seems to be overfitting and your model is not learning. Try SGD optimizer with a learning rate of 0.001 ADAM optimizer will give you a soon overfitting, and decreasing the learning rate will train ...
user avatar
  • 120
1 vote

How to use pretrained checkpoints of BERT model on semantic text similarity task?

Have a look at https://medium.com/the-artificial-impostor/news-topic-similarity-measure-using-pretrained-bert-model-1dbfe6a66f1d You can have the two sentences as first and second use the next ...
user avatar
1 vote

Will BERT embedding be always same for a given document when used as a feature extractor

BERT is deterministic. There is no variation unless you parse your tokens differently in succeeding runs. Here is the original paper the model architecture is based off of Transformer Paper. Note that ...
user avatar
  • 2,289

Only top scored, non community-wiki answers of a minimum length are eligible