Questions tagged [bert]
For questions related to BERT (which stands for Bidirectional Encoder Representations from Transformers), a language representation model introduced in the paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2019) by Google.
95 questions
1
vote
1
answer
18
views
How does DistilBERT's training ensure that the [CLS] token's hidden state from DistilBERT aligns with that from BERT?
BERT's pre-training involves next sentence prediction (NSP) based on a classifier on top of the [CLS] token's last hidden state. This is primarily what gives the [CLS] token's hidden state the special ...
0
votes
1
answer
32
views
Cosine embedding loss in DistilBERT
Does the cosine embedding loss used when pre-training DistilBERT align all hidden states output by DistilBERT with those output by BERT? Or, does it align only the hidden states of the masked tokens?
...
0
votes
0
answers
21
views
fine tuning Marbert for Tunisian dialect, ask for the tokenizer
i want to fine tune Marbert for tunisian text classification dialect, using this dataset : https://www.kaggle.com/datasets/waalbannyantudre/tunisian-arabizi-dialect-data-sentiment-analysis
i have test ...
-2
votes
0
answers
14
views
Text dump to table
Consider we parse an excel sheet and now have all the text from the sheet, this is a sort of a text dump, just a bag of words. Doing this we have lost information like headers, separators and what ...
- 1
0
votes
0
answers
13
views
How to handle empty input for RoBERTa text classification model?
I'm working on a text classification task using a fine-tuned RoBERTa model. My model takes text inputs and classifies them into predefined categories. However, I'm unsure how to handle cases where the ...
0
votes
0
answers
21
views
Correct way for handling padding masks in Transformer networks
In many NLP tasks, to handle batched processing, we pad the inputs in a given batch to match the length of the longest element in the batch. We also get a corresponding mask. Now, my point of ...
- 1
1
vote
1
answer
39
views
0
votes
0
answers
34
views
How can I change the tokens BERT uses so that each digit is a separate token?
Rather than have the tokenizer generate this sort of thing:
"$1009 Dollars" => ["$", "100#", "9", "Dollars"]
I'...
- 101
0
votes
1
answer
84
views
How do transformer models handle negation in sentiment analysis
I'm trying to understand how transformer models, such as BERT or GPT, handle negation in sentiment analysis. Specifically, I'm curious about how these models manage to correctly interpret sentences ...
- 101
0
votes
1
answer
69
views
How is the bidirectional context achieved in BERT?
I have read the paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" by Jacob Devlin et al. (2018) and "Improving Language Understanding by ...
- 3
0
votes
0
answers
37
views
How does BERT know how to where to add segment embeddings (i.e. to differentiate between two sentences packed in a single token sequence)
In addition to using a special [SEP] token to distinguish between two sentences, I understand that BERT also adds special learned embeddings to each sentence:
"we add a learned embedding to every ...
- 1
0
votes
0
answers
44
views
Is token mask masked in attention of encoders of bert?
I have recently researched on Bert structure. And the paper says we will mask some token at the input in 80%, 10% input be changed and 10% left remained. But I wonder if the mask token in the input be ...
- 1
3
votes
1
answer
831
views
why we use learnable positional encoding instead of Sinusoidal positional encoding
In the original paper of transformers they using positional encoding to capture the position of each word in the sentence and for calculate that it using sin and cos ,like shom in the image. In Bert ...
0
votes
1
answer
948
views
2
votes
1
answer
56
views
What does Figure 3 in the BERT paper represent?
The BERT paper has the following diagram (Figure 3):
It's captioned "Differences in pre-training model architectures".
However, I thought the BERT architecture was just a stack of attention ...
- 121
1
vote
1
answer
88
views
Is it possible to convert BERT embeddings to textual format?
There is a need of computation that needs to take place, although the person who is sending the raw data is concerned about privacy of such a data.
...
0
votes
0
answers
90
views
Which input embeddings are learned during pre-training in BERT? What about during fine-tuning?
I was reading the 2019 BERT paper and they mention how they use wordpieces that are then represented as the sum of token embeddings, segment embeddings, and positional embeddings. What is unclear to ...
- 1
0
votes
0
answers
519
views
What is the loss function used when pre-training BERT on MLM & NSP tasks?
I'm new to NLP and was reading through the 2019 BERT paper and am confused about the loss function used during pre-training.
As I understand it, the model is trained on the MLM and NSP tasks. The MLM ...
- 1
0
votes
1
answer
174
views
Fine Tuning a Bert Transformer. How to label for emotions and train large scripts?
From what I have seen you can fine tune a Bert model to detect emotions by labelling single sentences.
But if the text you want to evaluate is a large script with many sentences, do I need to split ...
- 53
0
votes
1
answer
410
views
How can BERT/Transformer models accept input batches of different sizes?
I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input.
How is that possible? I thought we ...
- 101
0
votes
2
answers
832
views
How big the context can be using HuggingFace models?
I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically ...
- 103
1
vote
0
answers
1k
views
What is MLM & NSP loss function
Two objective functions are used during the BERT language
model pretraining step.
The first one is masked language
model (MLM) that randomly masks
15% of the
input tokens and the objective is to ...
- 121
0
votes
1
answer
129
views
How to generate a sentence containing a specific set of tokens using GPT2 or BERT?
I have different sets of words as inputs, e.g.,
{governor, John Gary Evans, office, 1894}
or
{cheetah, 80km/h, mammal}
I would ...
- 51
0
votes
1
answer
81
views
How can I not only classify an intent, but also identify slots and values in it?
I've been working on text -> intent -> command execution for a particular application and while I've found many papers and code that work well for intent classification (1, 2, etc.), they stop ...
- 101
0
votes
1
answer
597
views
Multilabel text classification with highly imbalanced training data
I'm trying to train a multilabel text classification model using BERT. Each piece of text can belong to 0 or more of a total of 485 classes. My model consists of a dropout layer and a linear layer ...
0
votes
2
answers
55
views
How are gradients backpropogated in ALBERT?
I was reading the ALBERT paper and saw that they use the same parameters in each layer hence reducing the number of unique parameters. From what I could gather it seems if the all the layers have say ...
0
votes
1
answer
42
views
How to classify data into organised groups by using a resulting classification vector and a set of probabilities? [closed]
I am trying to figure out the best way to calculate the probability a sentence belongs to some category.
For simplicity sake, lets assume that the sentence is "yellow fruit".
Next, I use the ...
- 135
1
vote
0
answers
408
views
Left-to-Right vs Encoder-decoder Models
Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS)
Left-to-Right:
Auto-regressive, Left-to-right models, predict the ...
- 39
0
votes
2
answers
1k
views
How to combine pretrained language models with additional feature set?
Are there any techniques to combine a feature set (other than the text itself) with pretrained language models.
Let's say I have a random NLP task that tries to predict a binary class label based on e....
- 101
2
votes
1
answer
5k
views
Which positional encoding BERT use?
It is a little bit confusing that someone is explaining that BERT is using sinusoidal functions for BERT position encoding and someone is saying BERT just uses absolute position.
I checked that ...
- 121
2
votes
2
answers
9k
views
What is the loss function and training task on which the original BERT model was trained
I was checking on sentence embeddings and stumbled across the BERT model which employs transformers.
I understand that BERT applies a WordPice tokenizer (e.g. working like https://keras.io/api/...
- 188
1
vote
1
answer
672
views
How to Train a Decoder for Pre-trained BERT Transformer-Encoder?
Context:
I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings....
- 11
1
vote
1
answer
123
views
fondamental question about regularization techniques to solve overfitting problem in neural networks
I have a text classification neural network based on BERT that overfits. The accuracy on the training dataset is 95%, whereas it is 68% on the validation dataset.
Using some classical regularization ...
- 113
0
votes
1
answer
384
views
Hot to calculate Maximum Normalized log Probability for Active Learning with BERT
I have encountered difficulties understanding the calculation of Maximum Normalized Log Probabilities acording to Shen et al..
With n being the sequence length, yi the label of word i. Xij is the ...
- 1
0
votes
1
answer
55
views
Can I use Sentence-Bert to embed event triples?
I extracted event triples from sentences using OpenIE. Can I concatenate the components in the event triple to make it a sentence and use Sentence-Bert to embed the event?
It seems no one has done ...
- 121
0
votes
2
answers
945
views
Does it make sense to add an additional attention layer while fine-tuning Bert?
I'm fine tuning a Bert/Roberta model for a classification task.
As I need to improve my results, I'm thinking about to add an additional attention layer after Bert model and before dense and dropout ...
0
votes
0
answers
514
views
Next Sentence Prediction for 5 sentences using BERT
I am given a dataset in which each instance consisting of 5 sentences. The goal is to predict the sequence of numbers which represent the order of these sentences.
For example, given a story:
He went ...
0
votes
1
answer
2k
views
How do we reduce the output dimensions of BERT?
The output dimensions of BERT are 768-dimensional, is it possible to reduce them to a lower, custom number? For example, if there is another BERT-based transformer model which has a lower count of ...
- 1
0
votes
0
answers
170
views
Why is BERT/GPT capable of "for-all" generalization?
As shown in the figure:
Why does token prediction work when "Socrates" is replaced with "Plato"?
From the point of view of symbolic logic, the above example effectively performs ...
- 245
0
votes
0
answers
259
views
Fine tuning BERT for token level classification
I want to try self-supervised and semi-supervised learning for my task, which relates to token-wise classification for the 2 sequences of sentences (source and translated text). The labels would be ...
5
votes
3
answers
2k
views
What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?
In PyTorch, transformer (BERT) models have an intermediate dense layer in between attention and output layers whereas the BERT and Transformer papers just mention the attention connected directly to ...
1
vote
0
answers
128
views
How to fine-tune a model which was pre-trained on a corpus that contains words with different meanings than the meanings of those words on my corpus?
I have a scenario in which we should leverage previously asked questions (not questions pairs, single question in a column) to locate similar questions within those questions.
How can I fine-tune my ...
- 11
0
votes
1
answer
197
views
Training and Evaluating BERT and XLNET [closed]
I am thinking about a project and have a few questions before I accept it. Would be grateful I anyone experienced of you could give me some advice.
In the project, I have been given a data set with (...
- 19
3
votes
3
answers
9k
views
Isn't attention mask for BERT model useless?
I have just dived into deep learning for NLP, and now I'm learning how the BERT model works. What I found odd is why the BERT model needs to have an attention mask. As clearly shown in this tutorial ...
- 153
2
votes
1
answer
3k
views
Does BERT freeze the entire model body when it does fine-tuning?
Recently, I came across the BERT model. I did some research and tried some implementations.
I wanted to tackle a NER task, so I chose the BertForSequenceClassifications provided by HuggingFace.
...
- 51
2
votes
1
answer
106
views
Why are BERT embeddings interpreted as representations of the corresponding words?
It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a ...
2
votes
0
answers
29
views
Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)
I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:
...
2
votes
0
answers
109
views
What's new in LaBSE v2?
I can't find what's new in LaBSE v2 (https://tfhub.dev/google/LaBSE/2). What are the main highlights of v2 versus v1? And how did you find out?
- 121
1
vote
1
answer
327
views
Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?
Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
- 119
20
votes
2
answers
14k
views
Why does GPT-2 Exclude the Transformer Encoder?
After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
- 351