Questions tagged [bert]

For questions related to BERT (which stands for Bidirectional Encoder Representations from Transformers), a language representation model introduced in the paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2019) by Google.

Filter by
Sorted by
Tagged with
0
votes
0answers
22 views

Isn't attention mask for BERT model useless?

I have just dived into deep learning for NLP, and now I'm learning how the BERT model works. What I found odd is why the BERT model needs to have an attention mask. As clearly shown in this tutorial ...
0
votes
1answer
23 views

Does BERT freeze the entire model body when it does fine-tuning?

Recently, I came across the BERT model. I did some research and tried some implementations. I wanted to tackle a NER task, so I chose the BertForSequenceClassifications provided by HuggingFace. ...
0
votes
0answers
12 views

How many MAC operations are executed in one inference/training cycle of Google BERT?

I wonder if there is any information about the amount of MACs are executed for one training/inference cycle of Google BERT. I only found information about the number of layers and parameters here. ...
0
votes
0answers
20 views

Fine tuning a BERT model for text classification

An article written by Jay Alammar (http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/) on using a BERT transformer for text classification. The article mentions the following ...
1
vote
0answers
44 views

How can I add a Sequential CNN layer on top of BERT model?

Information I'm working on a binary classification task and used BERT model from transformers library to do it using the custom model below: ...
1
vote
0answers
20 views

Why are BERT embeddings interpreted as representations of the corresponding words?

It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a ...
0
votes
0answers
11 views

How can I prune BERT layers

I would like to finetune BERT on SQuAD and then evaluate the output from each layer (so from using 1 layer to using all 12). I know you can prune heads using Huggingface but was wondering how could ...
2
votes
0answers
15 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
1
vote
0answers
18 views

What's new in LaBSE v2?

I can't find what's new in LaBSE v2 (https://tfhub.dev/google/LaBSE/2). What are the main highlights of v2 versus v1? And how did you find out?
0
votes
0answers
8 views

What is the recommended way to retokenize ontonotes to wordpiece?

I am using ontonotes5 dataset which is already tokenize. But since I want to use Bert. I want to tokenize it myself, keeping all the annotations correct.
0
votes
1answer
38 views

Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?

Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
2
votes
2answers
114 views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
0
votes
0answers
16 views

Adding BERT embeddings in BiLSTM embedding layer

I am want to use BERT embeddings in the BiLSTM embedding layer instead of Word2Vec or FastText Embeddings. There is any code to do that?
0
votes
1answer
50 views

Should I need to use BERT embeddings while tokenizing using BERT tokenizer?

I am new to BERT and NLP and I am a little confused with tokenization and word embedding. My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use ...
0
votes
0answers
19 views

How does BERT answer questions?

I have been trying to understand how the BERT model works. Specifically, I am trying to understand how it picks up answers to questions on a given passage. I have tried following this blog post and, ...
1
vote
1answer
508 views

How do I calculate the probabilities of the BERT model prediction logits?

I might be getting this completely wrong, but please let me first try to explain what I need, and then what's wrong. I have a classification task. The training data has 50 different labels. The ...
0
votes
0answers
39 views

Understanding Huggingface QA Transformers Confidence Scores

I seem to be unable to find information on whether the confidence scores for the huggingface transformers used for QA are in the form of 10^-2 or 10^-5. What are your thoughts on the matter? Example: <...
2
votes
0answers
22 views

Adding corpus to BERT for QA

I was wondering about SciBERT's QA abilities using SQuAD. I have a scarce textual dataset consisting of less than 100 files where doctors are discussing cancer in dialogues. I want to add it to ...
0
votes
1answer
31 views

Sentiment analysis does not handle neturals [closed]

I'm writing some financial tools, I've found highly performant models for question and answering but when it comes to sentiment analysis I haven't found anything that good. I'm trying to use ...
1
vote
1answer
33 views

How to keep track of the subject/entity in a sentence?

I'm working on Sentiment Analysis, using HuggingFace to perform sentiment analysis on articles ...
1
vote
0answers
28 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
0
votes
0answers
18 views

How to chose a specific contextual embedding?

I am learning deep learning using a plethora of online resources. I have been using mostly word2vec and gloVe for my NMT model. I recently came across the concept of contextual-embedding, and I see ...
0
votes
0answers
16 views

Is parameter sharing in AlBERT akin to repeated application of same function on input?

I read the AlBERT and saw that its architecture used "Parameter Sharing" among layers of the encoder. They mentioned that this was done to save model space, make fewer training parameters ...
2
votes
0answers
92 views

T5 or BERT for sentence correction/generation task?

I have sentences with some grammatical errors , with no punctuations and digits written in words... something like below: As you can observe, a proper noun , winston isnt highlighted with capital in ...
1
vote
0answers
190 views

What is MNLI-(m/mm)?

I came across the term MNLI-(m/mm) in Table 1 of the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I know what MNLI stands for, i.e. Multi-Genre Natural ...
2
votes
1answer
72 views

Transformers: how to get the output (keys and values) of the encoder?

I was reading the paper Attention Is All You Need. It seems like the last step of the encoder is a LayerNorm(relu(WX + B) + X), i.e. an add + normalization. This should result in a $n$ x $d^{model}$ ...
2
votes
1answer
45 views

Transformers: how does the decoder final layer output the desired token?

In the paper Attention Is All You Need, this section confuses me: In our model, we share the same weight matrix between the two embedding layers [in the encoding section] and the pre-softmax linear ...
0
votes
0answers
27 views

Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?

I know that several tokenization methods that are used for tranformer models like WordPiece for Bert and BPE for Roberta and others. What I was wondering if there is also a transformer which uses a ...
1
vote
1answer
230 views

Is there a way to provide multiple masks to BERT in MLM task?

I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence. ...
0
votes
0answers
21 views

What BERT predicts when the token supposed to be masked is not masked?

I am reading the BERT paper. In the paper, they say that: Although this allows us to obtain a bidirec- tional pre-trained model, a downside is that we are creating a mismatch between pre-training and ...
2
votes
0answers
406 views

Why aren't the BERT layers frozen during fine-tuning tasks?

During transfer learning in computer vision, I've seen that the layers of the base model are frozen if the images aren't too different from the model on which the base model is trained on. However, on ...
5
votes
0answers
53 views

Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?

BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
0
votes
1answer
43 views

BERT: After pretraining 880000 step, why fine-tune not work? [closed]

I am using pretraining code from https://github.com/NVIDIA/DeepLearningExamples Pretrain parameters: ...
2
votes
0answers
39 views

Bert for Sentiment Analysis - Connecting final output back to the input

I have not found a lot of information on this, but I am wondering if there is a standard way to apply the outputs of a Bert model being used for sentiment analysis, and connect them back to the ...
0
votes
0answers
16 views

How to use an image tensor for caption generation with Transformer-XL or BERT?

I am fairly new to transformers and deep learning in general so please be kind, I am currently working on a project that will caption images using either Transformer-XL or BERT, however, I am not sure ...
5
votes
1answer
2k views

How is BERT different from the original transformer architecture?

As far as I can tell, BERT is a type of Transformer architecture. What I do not understand is: How is Bert different from the original transformer architecture? What tasks are better suited for BERT,...
0
votes
0answers
23 views

Why BERT last 4 Layers should be considered to extract word embeddings?

In most of the cases the embedding vectors of last 4 layers of BERT are summed up to represent the tokens embedding. I've tried to explore but haven't found any strong reason/resource on why we should ...
0
votes
0answers
13 views

What is the best NLP model for handeling a rapid extension of the production data for a question answeing task

I'm lately writing a chat-bot to answer questions about children's fantasy books I save to my database . When a user opens a book it loads the chat-bot for the given book . I don't need it to work on ...
0
votes
0answers
24 views

Automated Scoring (non-english language) Using BERT

i'm a student and i'm new to NLP. I want to build an Automated Scoring system which is in Indonesian Language using BERT. The system is expected to be able to measure the similarity of an answer(e.g: ...
4
votes
1answer
2k views

How to fine tune BERT for question answering?

I wish to train two domain-specific models: Domain 1: Constitution and related Legal Documents Domain 2: Technical and related documents. For Domain 1, I've access to a text-corpus with texts from ...
1
vote
1answer
163 views

How to use speaker's information as well as text for fine-tuning BERT?

I want to classify my corporate chat messages into a few categories such as question, answer, and report. I used a fine-tuned BERT model, and the result wasn't bad. Now, I started thinking about ways ...
1
vote
1answer
182 views

Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?

I was wondering if the BART or T5 models can do the task of generating sentences in English. Most of the models I have mentioned ...
0
votes
1answer
70 views

Why does the BERT NSP head linear layer have two outputs?

Here's the code in question. https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491 ...
1
vote
0answers
33 views

Two questions about the architecture of Google Bert model (in particular about parameters)

I'm looking for someone who can help me clarify a few details regarding the architecture of Bert model. Those details are necessary for me to come with a full understanding of Bert model, so your help ...
1
vote
0answers
24 views

Is the number of bidirectional LSTMs in seq2seq model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
0
votes
1answer
58 views

How to add a pretrained model to my layers to get embeddings?

I want to use a pretrained model found in [BERT Embeddings] https://github.com/UKPLab/sentence-transformers and I want to add a layer to get the sentence embeddings from the model and pass on to the ...
2
votes
1answer
254 views

Similarity score between 2 words using Pre-trained BERT using Pytorch

I'm trying to compare Glove, Fasttext, Bert on the basis of similarity between 2 words using Pre-trained Models. Glove and Fasttext had pre-trained models that could easily be used with gensim ...
1
vote
1answer
3k views

How to use pre-trained BERT to extract the vectors from sentences?

I'm trying to extract the vectors from the sentences. Spent soo much time searching for the pre-trained BERT models but found nothing. Is it possible to get the vectors using pre-trained BERT from ...
8
votes
1answer
837 views

What is the intuition behind the dot product attention?

I am watching Attention all you need, In that what is the intuition behind the dot product attention? $$A(q,K, V) = \sum_i\frac{e^{q.k_i}}{\sum_j e^{q.k_j}} v_i$$ becomes: $$A(Q,K, V) = softmax(QK^...
2
votes
0answers
23 views

Will structured knowledge bases continue to be used in question answering with the likes of BERT gaining popularity?

This may come across as an open and opinion-based question, I definitely want to hear expert opinions on the subject, but I am also looking for references to materials that I can read deeply. One of ...