Questions tagged [bert]
For questions related to BERT (which stands for Bidirectional Encoder Representations from Transformers), a language representation model introduced in the paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2019) by Google.
94 questions
2
votes
1
answer
9k
views
How do I calculate the probabilities of the BERT model prediction logits?
I might be getting this completely wrong, but please let me first try to explain what I need, and then what's wrong.
I have a classification task. The training data has 50 different labels. The ...
- 123
2
votes
0
answers
39
views
Adding corpus to BERT for QA
I was wondering about SciBERT's QA abilities using SQuAD. I have a scarce textual dataset consisting of less than 100 files where doctors are discussing cancer in dialogues. I want to add it to ...
- 121
0
votes
1
answer
90
views
Sentiment analysis does not handle neturals [closed]
I'm writing some financial tools, I've found highly performant models for question and answering but when it comes to sentiment analysis I haven't found anything that good. I'm trying to use ...
- 115
1
vote
1
answer
405
views
How to keep track of the subject/entity in a sentence?
I'm working on Sentiment Analysis, using HuggingFace to perform sentiment analysis on articles
...
- 115
1
vote
0
answers
666
views
What does the outputlayer of BERT for masked language modelling look like?
In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows:
In technical terms, the prediction of the output words ...
- 635
3
votes
0
answers
487
views
T5 or BERT for sentence correction/generation task?
I have sentences with some grammatical errors , with no punctuations and digits written in words... something like below:
As you can observe, a proper noun , winston isnt highlighted with capital in ...
- 131
1
vote
1
answer
3k
views
What is MNLI-(m/mm)?
I came across the term MNLI-(m/mm) in Table 1 of the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I know what MNLI stands for, i.e. Multi-Genre Natural ...
2
votes
1
answer
754
views
Transformers: how to get the output (keys and values) of the encoder?
I was reading the paper Attention Is All You Need.
It seems like the last step of the encoder is a LayerNorm(relu(WX + B) + X), i.e. an add + normalization. This should result in a $n$ x $d^{model}$ ...
- 1,690
5
votes
2
answers
1k
views
Transformers: how does the decoder final layer output the desired token?
In the paper Attention Is All You Need, this section confuses me:
In our model, we share the same weight matrix between the two embedding layers [in the encoding section] and the pre-softmax linear ...
- 1,690
4
votes
1
answer
452
views
Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?
I know that several tokenization methods that are used for tranformer models like WordPiece for Bert and BPE for Roberta and others. What I was wondering if there is also a transformer which uses a ...
- 41
1
vote
1
answer
2k
views
Is there a way to provide multiple masks to BERT in MLM task?
I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence.
...
- 317
4
votes
1
answer
2k
views
Why aren't the BERT layers frozen during fine-tuning tasks?
During transfer learning in computer vision, I've seen that the layers of the base model are frozen if the images aren't too different from the model on which the base model is trained on.
However, on ...
- 141
8
votes
2
answers
985
views
Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?
BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
- 313
0
votes
1
answer
67
views
BERT: After pretraining 880000 step, why fine-tune not work? [closed]
I am using pretraining code from https://github.com/NVIDIA/DeepLearningExamples
Pretrain parameters:
...
- 101
2
votes
0
answers
45
views
Bert for Sentiment Analysis - Connecting final output back to the input
I have not found a lot of information on this, but I am wondering if there is a standard way to apply the outputs of a Bert model being used for sentiment analysis, and connect them back to the ...
- 21
30
votes
1
answer
18k
views
How is BERT different from the original transformer architecture?
As far as I can tell, BERT is a type of Transformer architecture. What I do not understand is:
How is Bert different from the original transformer architecture?
What tasks are better suited for BERT,...
- 3,000
4
votes
1
answer
4k
views
How to fine tune BERT for question answering?
I wish to train two domain-specific models:
Domain 1: Constitution and related Legal Documents
Domain 2: Technical and related documents.
For Domain 1, I've access to a text-corpus with texts from ...
- 141
1
vote
1
answer
346
views
How to use speaker's information as well as text for fine-tuning BERT?
I want to classify my corporate chat messages into a few categories such as question, answer, and report. I used a fine-tuned BERT model, and the result wasn't bad. Now, I started thinking about ways ...
- 111
2
votes
1
answer
687
views
Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?
I was wondering if the BERT or T5 models can do the task of generating sentences in English. Most of the models I have mentioned ...
- 263
1
vote
1
answer
336
views
Why does the BERT NSP head linear layer have two outputs?
Here's the code in question.
https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491
...
- 197
1
vote
0
answers
42
views
Two questions about the architecture of Google Bert model (in particular about parameters)
I'm looking for someone who can help me clarify a few details regarding the architecture of Bert model. Those details are necessary for me to come with a full understanding of Bert model, so your help ...
- 11
1
vote
0
answers
75
views
Is the number of bidirectional LSTMs in seq2seq model equal to the maximum length of input text/characters?
I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/.
It ...
- 181
0
votes
1
answer
233
views
How to add a pretrained model to my layers to get embeddings?
I want to use a pretrained model found in [BERT Embeddings] https://github.com/UKPLab/sentence-transformers and I want to add a layer to get the sentence embeddings from the model and pass on to the ...
- 41
2
votes
1
answer
1k
views
Similarity score between 2 words using Pre-trained BERT using Pytorch
I'm trying to compare Glove, Fasttext, Bert on the basis of similarity between 2 words using Pre-trained Models. Glove and Fasttext had pre-trained models that could easily be used with gensim ...
- 1,283
2
votes
1
answer
6k
views
How to use pre-trained BERT to extract the vectors from sentences?
I'm trying to extract the vectors from the sentences. Spent soo much time searching for the pre-trained BERT models but found nothing.
Is it possible to get the vectors using pre-trained BERT from ...
- 1,283
22
votes
1
answer
12k
views
What is the intuition behind the dot product attention?
I am watching the video Attention Is All You Need by Yannic Kilcher.
My question is: what is the intuition behind the dot product attention?
$$A(q,K, V) = \sum_i\frac{e^{q.k_i}}{\sum_j e^{q.k_j}} v_i$$...
- 1,763
2
votes
0
answers
36
views
Will structured knowledge bases continue to be used in question answering with the likes of BERT gaining popularity?
This may come across as an open and opinion-based question, I definitely want to hear expert opinions on the subject, but I am also looking for references to materials that I can read deeply.
One of ...
- 121
1
vote
0
answers
369
views
Building a template based NLG system to generate a report from data
I am a newbie to NLP and NLG. I am tasked to develop a system to generate a report based on a given data table. The structure of the report and the flow is predefined. I have researched on several ...
- 11
3
votes
0
answers
724
views
How does one continue the pre-training in BERT?
I need some help with continuing pre-training on Bert. I have a very specific vocabulary and lots of specific abbreviations at hand. I want to do an STS task. Let me specify my task: I have domain-...
- 31
2
votes
2
answers
3k
views
Why is my loss (binary cross entropy) converging on ~0.6? (Task: Natural Language Inference)
I’m trying to debug my neural network (BERT fine-tuning) trained for natural language inference with binary classification of either entailment or contradiction. I've trained it for 80 epochs and its ...
- 35
3
votes
0
answers
573
views
Can Bert be used to extract embedding for large categorical features?
I've lot of training data points (i.e in millions) and I've around few features but the issue with that is all the features are categorical data with 1 million+ categories in each.
So, I couldn't use ...
- 149
7
votes
1
answer
1k
views
How to use BERT as a multi-purpose conversational AI?
I'm looking to make an NLP model that can achieve a dual purpose. One purpose is that it can hold interesting conversations (conversational AI), and another being that it can do intent classification ...
- 323
2
votes
1
answer
233
views
How to use pretrained checkpoints of BERT model on semantic text similarity task?
I am unaware to use the derived checkpoints from pre-trained BERT model for the task of semantic text similarity.
...
- 21
2
votes
1
answer
9k
views
Adding BERT embeddings in LSTM embedding layer
I am planning to use BERT embeddings in the LSTM embedding layer instead of the usual Word2vec/Glove Embeddings. What are the possible ways to do that?
5
votes
1
answer
787
views
Will BERT embedding be always same for a given document when used as a feature extractor
When we use BERT embeddings for a classification task, would we get different embeddings every time we pass the same text through the BERT architecture? If yes, is it the right way to use the ...
1
vote
0
answers
496
views
How are the attention weights normalised in the transformer?
In the Transformer (adopted in BERT), we normalize the attention weights (dot product of keys and queries) using a softmax in the Scaled Dot-Product mechanism. It is unclear to me whether this ...
- 131
3
votes
1
answer
2k
views
Understanding how the loss was calculated for the SQuAD task in BERT paper
In the BERT paper, section 4.2 covers the SQuAD training.
From my understanding, there are two extra parameters trained, they are two vectors with the same dimension as the hidden size, so the same ...
- 197
3
votes
0
answers
274
views
How does bidirectional encoding allow the predicted word to indirectly "see itself"?
Before the release of BERT, we used to say that it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that's ...
- 589
2
votes
0
answers
80
views
How can I generate a document from a single word using GPT or BERT?
I have a dataset of 100000 documents each labelled with a topic to it. I want to create a model such that, given a topic, the model can generate a document from it.
I came across language models GPT,...
- 161
2
votes
2
answers
733
views
Is it a good idea to use BERT to answer a FAQ with semantic similarity?
I have been looking for BERT for many tasks. I would like to compare the performance to answer an FAQ, using BERT semantic similarity and BERT Q/A.
However, I'm not sure it is a good idea to use ...
- 161
4
votes
1
answer
1k
views
Why does the BERT encoder have an intermediate layer between the attention and neural network layers with a bigger output?
I am reading the BERT paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
As I look at the attention mechanism, I don't understand why in the BERT encoder we have ...
- 161
11
votes
2
answers
13k
views
What are the segment embeddings and position embeddings in BERT?
They only reference in the paper that the position embeddings are learned, which is different from what was done in ELMo.
ELMo paper - https://arxiv.org/pdf/1802.05365.pdf
BERT paper - https://arxiv....
- 163
33
votes
3
answers
44k
views
Can BERT be used for sentence generating tasks?
I am a new learner in NLP. I am interested in the sentence generating task. As far as I am concerned, one state-of-the-art method is the CharRNN, which uses RNN to generate a sequence of words.
...
- 433
5
votes
2
answers
2k
views
Where can I find pre-trained language models in English and German? [closed]
Where can I find (more) pre-trained language models? I am especially interested in neural network-based models for English and German.
I am aware only of Language Model on One Billion Word Benchmark ...
- 161