# Tag Info

Accepted

### Can BERT be used for sentence generating tasks?

For newbies, NO. Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its ...
• 501
Accepted

### How is BERT different from the original transformer architecture?

What is a transformer? The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-...
• 37.1k

### What is the intuition behind the dot product attention?

Let's start with a bit of notation and a couple of important clarifications. $\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word. \$\...
• 4,773

### Why does GPT-2 Exclude the Transformer Encoder?

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
• 1,004

### Can BERT be used for sentence generating tasks?

this experiment by Stephen Mayhew suggests that BERT is lousy at sequential text generation: http://mayhewsw.github.io/2019/01/16/can-bert-generate-text/ ...
• 181

### What are the segment embeddings and position embeddings in BERT?

Sentences (for those tasks such as NLI which take two sentences as input) are differentiated in two ways in BERT: First, a [SEP] token is put between them Second, ...
Accepted

### How to use pre-trained BERT to extract the vectors from sentences?

Did you mean: How do you use a pre-trained BERT model in a feature-based setting to get pre-trained word contextual embeddings? Here is the BERT paper. I highly recommend you read it. Firstly, ...
• 156

### What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?

Feedforward layer is an important part of the transformer architecture. Transformer architecture, in addition to the self-attention layer, that aggregates ...

### Why does GPT-2 Exclude the Transformer Encoder?

The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot ...
• 51
Accepted

### Transformers: how to get the output (keys and values) of the encoder?

I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py). It seems like an extra linear layer learns ...
• 1,400

### Adding BERT embeddings in LSTM embedding layer

Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead. ...
Accepted

### What are the segment embeddings and position embeddings in BERT?

These embeddings are nothing more than token embeddings. You just randomly initialize them, then use gradient descent to train them, just like what you do with token embeddings.
• 501

### Isn't attention mask for BERT model useless?

This is just an implementation issue. One reason is the Huggingface implementation (which is not the original implementation by Google) wants to strictly separate the tokenization from the modeling. ...
• 241

### Can BERT be used for sentence generating tasks?

No. Sentence generating is directly related to language modelling (given the previous words in the sentence, what is the next word). Because of bi-directionality of BERT, BERT cannot be used as a ...
• 371
Accepted

### Understanding how the loss was calculated for the SQuAD task in BERT paper

These answers are based on my personal understanding of Bert from both the paper and official_implementation, hope it will help: ...
• 499
Accepted

### Why is my loss (binary cross entropy) converging on ~0.6? (Task: Natural Language Inference)

It seems to be overfitting and your model is not learning. Try SGD optimizer with a learning rate of 0.001 ADAM optimizer will give you a soon overfitting, and decreasing the learning rate will train ...
• 140

### How to use BERT as a multi-purpose conversational AI?

Think of BERT (or similar models) as as good starting place for understanding context. A couple options to make BERT contextualize dialogue: Concatenate all messages with a seperator embedding and ...
• 2,339

### How to use speaker's information as well as text for fine-tuning BERT?

My answer assumes your fine-tuning architecture simply stacks a single fully-connected layer on top of the BERT [CLS] output, as in Figure 4b of the BERT paper. ...
• 346
Accepted

### Transformers: how does the decoder final layer output the desired token?

I found the answer by reading the paper referenced by that section, Using the output embedding to improve language models Based on this observation, we propose threeway weight tying (TWWT), where the ...
• 1,400

### Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?

There is a pre-trained language model called ProphetNet for sequence-to-sequence learning with a novel self-supervised objective called future n-gram prediction. https://github.com/microsoft/...
• 121
Accepted

### How do I calculate the probabilities of the BERT model prediction logits?

Your call to model.predict() is returning the logits for softmax. This is useful for training purposes. To get probabilties, you need to apply softmax on the logits....
• 26.6k
Accepted

### How to keep track of the subject/entity in a sentence?

What you're describing is known as coreference resolution. More specifically, this example is anaphora resolution. The short answer is that this is an open research question and there is no well-...
Accepted

### Does BERT freeze the entire model body when it does fine-tuning?

Taken directly from HuggingFace Note that if you are used to freezing the body of your pretrained model (like in computer vision) the above may seem a bit strange, as we are directly fine-tuning the ...
• 41
Accepted

### fondamental question about regularization techniques to solve overfitting problem in neural networks

Regularization techniques reduce overfitting. This is why they tend to reduce training accuracy when applied to a model: they prevent the model to learn noise from the training data. For the same ...
• 4,773
Accepted

### How to classify data into organised groups by using a resulting classification vector and a set of probabilities?

To transform your logits [-5,1,2] into probabilities you can use the softmax function. This would get you ...
• 882
1 vote

### Should I need to use BERT embeddings while tokenizing using BERT tokenizer?

Generally speaking, the power of BERT for applications like NER is that the authors (of whichever implementation you use) performed a large-scale pretraining effort to create the embeddings. You can ...
• 266
1 vote
Accepted

### Sentiment analysis does not handle neturals

Yes, there is. You can try Spacy. Here you go. ...
1 vote

### How to fine tune BERT for question answering?

The answer is yes but 'lightweight' will require a 'lightweight' model. Your application for 'domain one' is called open domain question answering (ODQA). Here is a demonstration of ODQA using BERT: ...
• 1,698
1 vote

### Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?

=> I don't have more ideas about BART and T5 Right now. but I had created a chatbot Based on GPT-2 Model-based on Microsoft DialoGPT. Which is fine-tuned on millions of parameters of Reddit. you can ...
• 355
1 vote
Accepted

### Why does the BERT NSP head linear layer have two outputs?

This seems to be inherited from the original Google implementation, which also uses 2 outputs (https://github.com/google-research/bert/blob/master/run_pretraining.py#L293). I can see two possible ...
• 346

Only top scored, non community-wiki answers of a minimum length are eligible