2022 Developer Survey is open! Take survey.

Questions tagged [bert]

For questions related to BERT (which stands for Bidirectional Encoder Representations from Transformers), a language representation model introduced in the paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2019) by Google.

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

What is the ideal GPU/CPU requirment for albert xxlargev1 squad2? [closed]

I am quite new to AI and Deep learning but I have specific question about what is the requiments to speed/create a conversational AI based on ALbert model. https://huggingface.co/ahotrod/...
user avatar
0 votes
1 answer
26 views

Does make sense to add an additional Attention layer while Fine-Tuning Bert?

I'm fine tuning a Bert/Roberta model for a classification task. As I need to improve my results I'm thinking about to add an additional Attention layer after Bert Model and before Dense and Dropout ...
user avatar
0 votes
0 answers
17 views

How to fine-tune RoBERTa using Triplet Objective Function

I want to see if we can improve the triplet objective function of SBERT by slightly tweaking the equation terms. To do so, In your opinion, what's the easiest way to fine-tune RoBERTa? How can I ...
user avatar
  • 103
0 votes
0 answers
9 views

What is the correct way to do BIO notation for labelling in named entity recognition?

I'm trying to understand, what is the correct approach for labelling for NER. Some sources state: use BIO notation, which differentiates the beginning (B) and the inside (I) of entities ...
user avatar
0 votes
0 answers
21 views

Modify the architecture of the pre-trained model

I'm trying to remove some layers and reduce the size of fully connected in pre-trained BERT model. And then I'm going to fine-tune the modified models on GLUE. But the problem is that the modified ...
user avatar
  • 1
0 votes
0 answers
59 views

Next Sentence Prediction for 5 sentences using BERT

I am given a dataset in which each instance consisting of 5 sentences. The goal is to predict the sequence of numbers which represent the order of these sentences. For example, given a story: He went ...
user avatar
0 votes
0 answers
123 views

How do we reduce the output dimensions of BERT?

The output dimensions of BERT are 768-dimensional, is it possible to reduce them to a lower, custom number? For example, if there is another BERT-based transformer model which has a lower count of ...
user avatar
0 votes
0 answers
42 views

Which deep learning architecture for NLP is most accurate (and at the same time easy to implement)?

I have created a data set with 30.000 text documents (each text file is rather small with respect to its length), which are labelled with 0 and 1. Using this data set, I want to train a deep learning ...
user avatar
0 votes
0 answers
13 views

XLMRoberta loss remains constant over iterations for TokenClassification task

I have created a simple XLMRoberta model for token classification. The task is to predict the quality of translation for each token/word. The data looks something like this, where the first sentence ...
user avatar
0 votes
0 answers
34 views

Why is BERT/GPT capable of "for-all" generalization?

As shown in the figure: Why does token prediction work when "Socrates" is replaced with "Plato"? From the point of view of symbolic logic, the above example effectively performs ...
user avatar
0 votes
0 answers
51 views

Fine tuning BERT for token level classification

I want to try self-supervised and semi-supervised learning for my task, which relates to token-wise classification for the 2 sequences of sentences (source and translated text). The labels would be ...
user avatar
3 votes
1 answer
139 views

What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?

In PyTorch, transformer (BERT) models have an intermediate dense layer in between attention and output layers whereas the BERT and Transformer papers just mention the attention connected directly to ...
user avatar
0 votes
0 answers
61 views

Bert vs Sentence-Bert

I read a paper about Rumor detection and they used BERT as an unsupervised language representation, fine-tuning it using a small dataset, and combining it with a supervised learning model to provide ...
user avatar
0 votes
0 answers
65 views

How to fine-tune a model which was pre-trained on a corpus that contains words with different meanings than the meanings of those words on my corpus?

I have a scenario in which we should leverage previously asked questions (not questions pairs, single question in a column) to locate similar questions within those questions. How can I fine-tune my ...
user avatar
  • 1
0 votes
1 answer
70 views

Training and Evaluating BERT and XLNET [closed]

I am thinking about a project and have a few questions before I accept it. Would be grateful I anyone experienced of you could give me some advice. In the project, I have been given a data set with (...
user avatar
1 vote
1 answer
1k views

Isn't attention mask for BERT model useless?

I have just dived into deep learning for NLP, and now I'm learning how the BERT model works. What I found odd is why the BERT model needs to have an attention mask. As clearly shown in this tutorial ...
user avatar
0 votes
1 answer
510 views

Does BERT freeze the entire model body when it does fine-tuning?

Recently, I came across the BERT model. I did some research and tried some implementations. I wanted to tackle a NER task, so I chose the BertForSequenceClassifications provided by HuggingFace. ...
user avatar
  • 1
0 votes
0 answers
61 views

How many MAC operations are executed in one inference/training cycle of Google BERT?

I wonder if there is any information about the amount of MACs are executed for one training/inference cycle of Google BERT. I only found information about the number of layers and parameters here. ...
user avatar
0 votes
0 answers
44 views

Fine tuning a BERT model for text classification

An article written by Jay Alammar (http://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/) on using a BERT transformer for text classification. The article mentions the following ...
user avatar
1 vote
0 answers
28 views

Why are BERT embeddings interpreted as representations of the corresponding words?

It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a ...
user avatar
2 votes
0 answers
21 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
user avatar
2 votes
0 answers
36 views

What's new in LaBSE v2?

I can't find what's new in LaBSE v2 (https://tfhub.dev/google/LaBSE/2). What are the main highlights of v2 versus v1? And how did you find out?
user avatar
  • 121
1 vote
1 answer
61 views

Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?

Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
user avatar
  • 119
5 votes
2 answers
958 views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
user avatar
1 vote
2 answers
294 views

Should I need to use BERT embeddings while tokenizing using BERT tokenizer?

I am new to BERT and NLP and I am a little confused with tokenization and word embedding. My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use ...
user avatar
1 vote
1 answer
2k views

How do I calculate the probabilities of the BERT model prediction logits?

I might be getting this completely wrong, but please let me first try to explain what I need, and then what's wrong. I have a classification task. The training data has 50 different labels. The ...
user avatar
  • 113
2 votes
0 answers
24 views

Adding corpus to BERT for QA

I was wondering about SciBERT's QA abilities using SQuAD. I have a scarce textual dataset consisting of less than 100 files where doctors are discussing cancer in dialogues. I want to add it to ...
user avatar
0 votes
1 answer
42 views

Sentiment analysis does not handle neturals [closed]

I'm writing some financial tools, I've found highly performant models for question and answering but when it comes to sentiment analysis I haven't found anything that good. I'm trying to use ...
user avatar
  • 115
1 vote
1 answer
45 views

How to keep track of the subject/entity in a sentence?

I'm working on Sentiment Analysis, using HuggingFace to perform sentiment analysis on articles ...
user avatar
  • 115
1 vote
0 answers
64 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
user avatar
2 votes
0 answers
170 views

T5 or BERT for sentence correction/generation task?

I have sentences with some grammatical errors , with no punctuations and digits written in words... something like below: As you can observe, a proper noun , winston isnt highlighted with capital in ...
user avatar
1 vote
1 answer
545 views

What is MNLI-(m/mm)?

I came across the term MNLI-(m/mm) in Table 1 of the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I know what MNLI stands for, i.e. Multi-Genre Natural ...
user avatar
2 votes
1 answer
172 views

Transformers: how to get the output (keys and values) of the encoder?

I was reading the paper Attention Is All You Need. It seems like the last step of the encoder is a LayerNorm(relu(WX + B) + X), i.e. an add + normalization. This should result in a $n$ x $d^{model}$ ...
user avatar
  • 1,210
3 votes
1 answer
124 views

Transformers: how does the decoder final layer output the desired token?

In the paper Attention Is All You Need, this section confuses me: In our model, we share the same weight matrix between the two embedding layers [in the encoding section] and the pre-softmax linear ...
user avatar
  • 1,210
2 votes
1 answer
133 views

Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?

I know that several tokenization methods that are used for tranformer models like WordPiece for Bert and BPE for Roberta and others. What I was wondering if there is also a transformer which uses a ...
user avatar
  • 21
1 vote
1 answer
568 views

Is there a way to provide multiple masks to BERT in MLM task?

I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence. ...
user avatar
2 votes
0 answers
764 views

Why aren't the BERT layers frozen during fine-tuning tasks?

During transfer learning in computer vision, I've seen that the layers of the base model are frozen if the images aren't too different from the model on which the base model is trained on. However, on ...
user avatar
6 votes
1 answer
112 views

Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?

BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
user avatar
0 votes
1 answer
49 views

BERT: After pretraining 880000 step, why fine-tune not work? [closed]

I am using pretraining code from https://github.com/NVIDIA/DeepLearningExamples Pretrain parameters: ...
user avatar
  • 101
2 votes
0 answers
41 views

Bert for Sentiment Analysis - Connecting final output back to the input

I have not found a lot of information on this, but I am wondering if there is a standard way to apply the outputs of a Bert model being used for sentiment analysis, and connect them back to the ...
user avatar
  • 21
15 votes
1 answer
6k views

How is BERT different from the original transformer architecture?

As far as I can tell, BERT is a type of Transformer architecture. What I do not understand is: How is Bert different from the original transformer architecture? What tasks are better suited for BERT,...
user avatar
4 votes
1 answer
3k views

How to fine tune BERT for question answering?

I wish to train two domain-specific models: Domain 1: Constitution and related Legal Documents Domain 2: Technical and related documents. For Domain 1, I've access to a text-corpus with texts from ...
user avatar
1 vote
1 answer
241 views

How to use speaker's information as well as text for fine-tuning BERT?

I want to classify my corporate chat messages into a few categories such as question, answer, and report. I used a fine-tuned BERT model, and the result wasn't bad. Now, I started thinking about ways ...
user avatar
  • 111
1 vote
1 answer
271 views

Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?

I was wondering if the BART or T5 models can do the task of generating sentences in English. Most of the models I have mentioned ...
user avatar
  • 233
1 vote
1 answer
161 views

Why does the BERT NSP head linear layer have two outputs?

Here's the code in question. https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491 ...
user avatar
1 vote
0 answers
37 views

Two questions about the architecture of Google Bert model (in particular about parameters)

I'm looking for someone who can help me clarify a few details regarding the architecture of Bert model. Those details are necessary for me to come with a full understanding of Bert model, so your help ...
user avatar
  • 11
1 vote
0 answers
25 views

Is the number of bidirectional LSTMs in seq2seq model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
user avatar
  • 151
0 votes
1 answer
109 views

How to add a pretrained model to my layers to get embeddings?

I want to use a pretrained model found in [BERT Embeddings] https://github.com/UKPLab/sentence-transformers and I want to add a layer to get the sentence embeddings from the model and pass on to the ...
user avatar
2 votes
1 answer
487 views

Similarity score between 2 words using Pre-trained BERT using Pytorch

I'm trying to compare Glove, Fasttext, Bert on the basis of similarity between 2 words using Pre-trained Models. Glove and Fasttext had pre-trained models that could easily be used with gensim ...
user avatar
  • 1,133
1 vote
1 answer
4k views

How to use pre-trained BERT to extract the vectors from sentences?

I'm trying to extract the vectors from the sentences. Spent soo much time searching for the pre-trained BERT models but found nothing. Is it possible to get the vectors using pre-trained BERT from ...
user avatar
  • 1,133