Skip to main content

Questions tagged [bert]

For questions related to BERT (which stands for Bidirectional Encoder Representations from Transformers), a language representation model introduced in the paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2019) by Google.

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Is token mask masked in attention of encoders of bert?

I have recently researched on Bert structure. And the paper says we will mask some token at the input in 80%, 10% input be changed and 10% left remained. But I wonder if the mask token in the input be ...
Thành An's user avatar
1 vote
1 answer
33 views

why we use learnable positional encoding instead of Sinusoidal positional encoding

In the original paper of transformers they using positional encoding to capture the position of each word in the sentence and for calculate that it using sin and cos ,like shom in the image. In Bert ...
LAILA EL OUEDEGHYRY's user avatar
-1 votes
1 answer
79 views

Why are some of the weights not initialized from the pretrained model checkpoint (from hugging face)? [closed]

...
Sebastian Nielsen's user avatar
2 votes
1 answer
44 views

What does Figure 3 in the BERT paper represent?

The BERT paper has the following diagram (Figure 3): It's captioned "Differences in pre-training model architectures". However, I thought the BERT architecture was just a stack of attention ...
statusfailed's user avatar
1 vote
1 answer
66 views

Is it possible to convert BERT embeddings to textual format?

There is a need of computation that needs to take place, although the person who is sending the raw data is concerned about privacy of such a data. ...
Vaibhav Maheshwari's user avatar
0 votes
0 answers
51 views

Which input embeddings are learned during pre-training in BERT? What about during fine-tuning?

I was reading the 2019 BERT paper and they mention how they use wordpieces that are then represented as the sum of token embeddings, segment embeddings, and positional embeddings. What is unclear to ...
Karla's user avatar
  • 1
0 votes
0 answers
188 views

What is the loss function used when pre-training BERT on MLM & NSP tasks?

I'm new to NLP and was reading through the 2019 BERT paper and am confused about the loss function used during pre-training. As I understand it, the model is trained on the MLM and NSP tasks. The MLM ...
Karla's user avatar
  • 1
0 votes
1 answer
137 views

Fine Tuning a Bert Transformer. How to label for emotions and train large scripts?

From what I have seen you can fine tune a Bert model to detect emotions by labelling single sentences. But if the text you want to evaluate is a large script with many sentences, do I need to split ...
arame3333's user avatar
0 votes
1 answer
169 views

How can BERT/Transformer models accept input batches of different sizes?

I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input. How is that possible? I thought we ...
PS1's user avatar
  • 101
0 votes
1 answer
574 views

How big the context can be using HuggingFace models?

I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically ...
Magno C's user avatar
  • 101
0 votes
0 answers
52 views

How to create dataset to extract information and classify intent using BERT?

Given a message: "Hey I am XYZ person (description about oneself), and I was thinking to launch a youtube video, wanted to get in touch with someone with similar experience", the model ...
thecalendar's user avatar
1 vote
0 answers
621 views

What is MLM & NSP loss function

Two objective functions are used during the BERT language model pretraining step. The first one is masked language model (MLM) that randomly masks 15% of the input tokens and the objective is to ...
XYZ's user avatar
  • 121
0 votes
1 answer
126 views

How to generate a sentence containing a specific set of tokens using GPT2 or BERT?

I have different sets of words as inputs, e.g., {governor, John Gary Evans, office, 1894} or {cheetah, 80km/h, mammal} I would ...
Vladimir's user avatar
0 votes
1 answer
77 views

How can I not only classify an intent, but also identify slots and values in it?

I've been working on text -> intent -> command execution for a particular application and while I've found many papers and code that work well for intent classification (1, 2, etc.), they stop ...
Ani's user avatar
  • 101
0 votes
1 answer
400 views

Multilabel text classification with highly imbalanced training data

I'm trying to train a multilabel text classification model using BERT. Each piece of text can belong to 0 or more of a total of 485 classes. My model consists of a dropout layer and a linear layer ...
Fijoy Vadakkumpadan's user avatar
0 votes
2 answers
52 views

How are gradients backpropogated in ALBERT?

I was reading the ALBERT paper and saw that they use the same parameters in each layer hence reducing the number of unique parameters. From what I could gather it seems if the all the layers have say ...
FoundABetterName's user avatar
0 votes
1 answer
38 views

How to classify data into organised groups by using a resulting classification vector and a set of probabilities? [closed]

I am trying to figure out the best way to calculate the probability a sentence belongs to some category. For simplicity sake, lets assume that the sentence is "yellow fruit". Next, I use the ...
Damir Olejar's user avatar
1 vote
0 answers
350 views

Left-to-Right vs Encoder-decoder Models

Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS) Left-to-Right: Auto-regressive, Left-to-right models, predict the ...
keyboardAnt's user avatar
0 votes
2 answers
1k views

How to combine pretrained language models with additional feature set?

Are there any techniques to combine a feature set (other than the text itself) with pretrained language models. Let's say I have a random NLP task that tries to predict a binary class label based on e....
fragant's user avatar
  • 101
2 votes
1 answer
4k views

Which positional encoding BERT use?

It is a little bit confusing that someone is explaining that BERT is using sinusoidal functions for BERT position encoding and someone is saying BERT just uses absolute position. I checked that ...
yoon's user avatar
  • 121
2 votes
2 answers
7k views

What is the loss function and training task on which the original BERT model was trained

I was checking on sentence embeddings and stumbled across the BERT model which employs transformers. I understand that BERT applies a WordPice tokenizer (e.g. working like https://keras.io/api/...
Ggjj11's user avatar
  • 188
1 vote
1 answer
457 views

How to Train a Decoder for Pre-trained BERT Transformer-Encoder?

Context: I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings....
node_env's user avatar
1 vote
1 answer
118 views

fondamental question about regularization techniques to solve overfitting problem in neural networks

I have a text classification neural network based on BERT that overfits. The accuracy on the training dataset is 95%, whereas it is 68% on the validation dataset. Using some classical regularization ...
tammuz's user avatar
  • 113
0 votes
1 answer
278 views

Hot to calculate Maximum Normalized log Probability for Active Learning with BERT

I have encountered difficulties understanding the calculation of Maximum Normalized Log Probabilities acording to Shen et al.. With n being the sequence length, yi the label of word i. Xij is the ...
Tobias H 's user avatar
0 votes
1 answer
46 views

Can I use Sentence-Bert to embed event triples?

I extracted event triples from sentences using OpenIE. Can I concatenate the components in the event triple to make it a sentence and use Sentence-Bert to embed the event? It seems no one has done ...
user900476's user avatar
0 votes
2 answers
772 views

Does it make sense to add an additional attention layer while fine-tuning Bert?

I'm fine tuning a Bert/Roberta model for a classification task. As I need to improve my results, I'm thinking about to add an additional attention layer after Bert model and before dense and dropout ...
Santiago Gonzalez Silot's user avatar
0 votes
0 answers
501 views

Next Sentence Prediction for 5 sentences using BERT

I am given a dataset in which each instance consisting of 5 sentences. The goal is to predict the sequence of numbers which represent the order of these sentences. For example, given a story: He went ...
lostpatriot's user avatar
0 votes
1 answer
2k views

How do we reduce the output dimensions of BERT?

The output dimensions of BERT are 768-dimensional, is it possible to reduce them to a lower, custom number? For example, if there is another BERT-based transformer model which has a lower count of ...
micawber's user avatar
0 votes
0 answers
158 views

Why is BERT/GPT capable of "for-all" generalization?

As shown in the figure: Why does token prediction work when "Socrates" is replaced with "Plato"? From the point of view of symbolic logic, the above example effectively performs ...
Yan King Yin's user avatar
0 votes
0 answers
241 views

Fine tuning BERT for token level classification

I want to try self-supervised and semi-supervised learning for my task, which relates to token-wise classification for the 2 sequences of sentences (source and translated text). The labels would be ...
Tarun Bhatia's user avatar
5 votes
3 answers
2k views

What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?

In PyTorch, transformer (BERT) models have an intermediate dense layer in between attention and output layers whereas the BERT and Transformer papers just mention the attention connected directly to ...
mohammad ali Humayun's user avatar
1 vote
0 answers
124 views

How to fine-tune a model which was pre-trained on a corpus that contains words with different meanings than the meanings of those words on my corpus?

I have a scenario in which we should leverage previously asked questions (not questions pairs, single question in a column) to locate similar questions within those questions. How can I fine-tune my ...
smanem's user avatar
  • 11
0 votes
1 answer
184 views

Training and Evaluating BERT and XLNET [closed]

I am thinking about a project and have a few questions before I accept it. Would be grateful I anyone experienced of you could give me some advice. In the project, I have been given a data set with (...
MiFischer22's user avatar
3 votes
3 answers
8k views

Isn't attention mask for BERT model useless?

I have just dived into deep learning for NLP, and now I'm learning how the BERT model works. What I found odd is why the BERT model needs to have an attention mask. As clearly shown in this tutorial ...
artas2357's user avatar
  • 153
2 votes
1 answer
3k views

Does BERT freeze the entire model body when it does fine-tuning?

Recently, I came across the BERT model. I did some research and tried some implementations. I wanted to tackle a NER task, so I chose the BertForSequenceClassifications provided by HuggingFace. ...
Joon's user avatar
  • 51
2 votes
1 answer
90 views

Why are BERT embeddings interpreted as representations of the corresponding words?

It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a ...
EmbeddingEnthusiast's user avatar
2 votes
0 answers
26 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
Bloodstone Programmer's user avatar
2 votes
0 answers
107 views

What's new in LaBSE v2?

I can't find what's new in LaBSE v2 (https://tfhub.dev/google/LaBSE/2). What are the main highlights of v2 versus v1? And how did you find out?
Brainless's user avatar
  • 121
1 vote
1 answer
289 views

Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?

Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
Nyxynyx's user avatar
  • 119
20 votes
2 answers
13k views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
Athena Wisdom's user avatar
1 vote
2 answers
1k views

Should I need to use BERT embeddings while tokenizing using BERT tokenizer?

I am new to BERT and NLP and I am a little confused with tokenization and word embedding. My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use ...
thenocturnalguy's user avatar
2 votes
1 answer
9k views

How do I calculate the probabilities of the BERT model prediction logits?

I might be getting this completely wrong, but please let me first try to explain what I need, and then what's wrong. I have a classification task. The training data has 50 different labels. The ...
iso_9001_'s user avatar
  • 123
2 votes
0 answers
38 views

Adding corpus to BERT for QA

I was wondering about SciBERT's QA abilities using SQuAD. I have a scarce textual dataset consisting of less than 100 files where doctors are discussing cancer in dialogues. I want to add it to ...
DarknessPlusPlus's user avatar
0 votes
1 answer
83 views

Sentiment analysis does not handle neturals [closed]

I'm writing some financial tools, I've found highly performant models for question and answering but when it comes to sentiment analysis I haven't found anything that good. I'm trying to use ...
johnny 5's user avatar
  • 115
1 vote
1 answer
325 views

How to keep track of the subject/entity in a sentence?

I'm working on Sentiment Analysis, using HuggingFace to perform sentiment analysis on articles ...
johnny 5's user avatar
  • 115
1 vote
0 answers
579 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
Bert Gayus's user avatar
3 votes
0 answers
459 views

T5 or BERT for sentence correction/generation task?

I have sentences with some grammatical errors , with no punctuations and digits written in words... something like below: As you can observe, a proper noun , winston isnt highlighted with capital in ...
Varun kadekar's user avatar
1 vote
1 answer
3k views

What is MNLI-(m/mm)?

I came across the term MNLI-(m/mm) in Table 1 of the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I know what MNLI stands for, i.e. Multi-Genre Natural ...
Marc-Philipp Knechtle's user avatar
2 votes
1 answer
683 views

Transformers: how to get the output (keys and values) of the encoder?

I was reading the paper Attention Is All You Need. It seems like the last step of the encoder is a LayerNorm(relu(WX + B) + X), i.e. an add + normalization. This should result in a $n$ x $d^{model}$ ...
user3667125's user avatar
  • 1,570
5 votes
2 answers
946 views

Transformers: how does the decoder final layer output the desired token?

In the paper Attention Is All You Need, this section confuses me: In our model, we share the same weight matrix between the two embedding layers [in the encoding section] and the pre-softmax linear ...
user3667125's user avatar
  • 1,570