Questions tagged [bert]
For questions related to BERT (which stands for Bidirectional Encoder Representations from Transformers), a language representation model introduced in the paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2019) by Google.
85
questions
0
votes
1
answer
64
views
Fine Tuning a Bert Transformer. How to label for emotions and train large scripts?
From what I have seen you can fine tune a Bert model to detect emotions by labelling single sentences.
But if the text you want to evaluate is a large script with many sentences, do I need to split ...
- 53
0
votes
1
answer
31
views
How can BERT/Transformer models accept input batches of different sizes?
I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input.
How is that possible? I thought we ...
- 1
0
votes
0
answers
141
views
How big the context can be using HuggingFace models?
I'm new on AI, Neural Networks, ChatBots and all this ecosystem. I'm trying to use a classical example of pre-trained models, more specifically ...
- 101
0
votes
0
answers
28
views
How to create dataset to extract information and classify intent using BERT?
Given a message: "Hey I am XYZ person (description about oneself), and I was thinking to launch a youtube video, wanted to get in touch with someone with similar experience", the model ...
1
vote
0
answers
198
views
What is MLM & NSP loss function
Two objective functions are used during the BERT language
model pretraining step.
The first one is masked language
model (MLM) that randomly masks
15% of the
input tokens and the objective is to ...
- 121
0
votes
1
answer
105
views
How to generate a sentence containing a specific set of tokens using GPT2 or BERT?
I have different sets of words as inputs, e.g.,
{governor, John Gary Evans, office, 1894}
or
{cheetah, 80km/h, mammal}
I would ...
- 51
0
votes
1
answer
66
views
How can I not only classify an intent, but also identify slots and values in it?
I've been working on text -> intent -> command execution for a particular application and while I've found many papers and code that work well for intent classification (1, 2, etc.), they stop ...
- 101
0
votes
0
answers
49
views
Can I use a pre-trained BERT to generate embeddings for training dataset then to fine tune the same BERT for semantic similarity?
I would like to fine-tune a sentence BERT model using my own dataset and perform a semantic similarity task. When generating the training dataset, I need to generate the embeddings for each sentence ...
- 101
0
votes
1
answer
222
views
Multilabel text classification with highly imbalanced training data
I'm trying to train a multilabel text classification model using BERT. Each piece of text can belong to 0 or more of a total of 485 classes. My model consists of a dropout layer and a linear layer ...
0
votes
0
answers
25
views
Tokenization for treelike structures
I'm pretraining a BERT (bigbird) model to use with SMILES encoding of chemicals. This kind of data is a treelike structure in the form of a string with a single bracket type. Usually this tree isn't ...
0
votes
0
answers
41
views
Transfer learning (or fine-tuning) a pre-trained model on multiple features?
I am currently fine-tuning a sentiment analysis bert-based model using PyTorch Trainer from hugging face. So far, so good.
I have easily managed to fine-tune the model on my text data. However, I'd ...
- 101
0
votes
2
answers
43
views
How are gradients backpropogated in ALBERT?
I was reading the ALBERT paper and saw that they use the same parameters in each layer hence reducing the number of unique parameters. From what I could gather it seems if the all the layers have say ...
0
votes
0
answers
243
views
Combining fine-tuning BERT and cross validation for hyperparameter selections
Is it possible to combine cross-validation procedure and hyper-parameter tuning for fine-tuning bert for a classification task?
The idea is the following:
Choose a set of set of hyperparameters {H,H1,...
- 113
0
votes
0
answers
29
views
How can you add data to BERT? Will 10-20 books added affect the word embeddings?
I will be using BERT to get word embeddings before performing cosine similarity analysis on my data. According to this paper the accuracy of word embeddings can be improved by updating the model with ...
- 13
0
votes
1
answer
36
views
How to classify data into organised groups by using a resulting classification vector and a set of probabilities? [closed]
I am trying to figure out the best way to calculate the probability a sentence belongs to some category.
For simplicity sake, lets assume that the sentence is "yellow fruit".
Next, I use the ...
- 135
0
votes
0
answers
64
views
How to fix the embedding gauge in a BERT-like model?
I have a pre-trained BERT model from Huggingface, which I tune to categorize short texts (like tweets or slightly longer) into several thousand categories using triplet loss.
As I understand, if I ...
- 101
0
votes
0
answers
130
views
How Could I fine tune bert-large-uncased-whole-word-masking or albert-xxlarge-v2 for question answering in Business, Financial domain for report?
I used mfeb/albert-xxlarge-v2-squad2,bert-large-uncased-whole-word-masking-finetuned-squad to answer the report question , but just get a little correct answer? Could I build a dataset like squad ...
1
vote
0
answers
257
views
Left-to-Right vs Encoder-decoder Models
Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS)
Left-to-Right:
Auto-regressive, Left-to-right models, predict the ...
- 39
0
votes
2
answers
835
views
How to combine pretrained language models with additional feature set?
Are there any techniques to combine a feature set (other than the text itself) with pretrained language models.
Let's say I have a random NLP task that tries to predict a binary class label based on e....
- 101
1
vote
1
answer
2k
views
Which positional encoding BERT use?
It is a little bit confusing that someone is explaining that BERT is using sinusoidal functions for BERT position encoding and someone is saying BERT just uses absolute position.
I checked that ...
- 111
1
vote
1
answer
5k
views
What is the loss function and training task on which the original BERT model was trained
I was checking on sentence embeddings and stumbled across the BERT model which employs transformers.
I understand that BERT applies a WordPice tokenizer (e.g. working like https://keras.io/api/...
- 178
1
vote
1
answer
264
views
How to Train a Decoder for Pre-trained BERT Transformer-Encoder?
Context:
I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings....
- 11
1
vote
1
answer
103
views
fondamental question about regularization techniques to solve overfitting problem in neural networks
I have a text classification neural network based on BERT that overfits. The accuracy on the training dataset is 95%, whereas it is 68% on the validation dataset.
Using some classical regularization ...
- 113
0
votes
1
answer
218
views
Hot to calculate Maximum Normalized log Probability for Active Learning with BERT
I have encountered difficulties understanding the calculation of Maximum Normalized Log Probabilities acording to Shen et al..
With n being the sequence length, yi the label of word i. Xij is the ...
- 1
0
votes
1
answer
40
views
Can I use Sentence-Bert to embed event triples?
I extracted event triples from sentences using OpenIE. Can I concatenate the components in the event triple to make it a sentence and use Sentence-Bert to embed the event?
It seems no one has done ...
- 121
0
votes
1
answer
617
views
Does it make sense to add an additional attention layer while fine-tuning Bert?
I'm fine tuning a Bert/Roberta model for a classification task.
As I need to improve my results, I'm thinking about to add an additional attention layer after Bert model and before dense and dropout ...
0
votes
0
answers
479
views
Next Sentence Prediction for 5 sentences using BERT
I am given a dataset in which each instance consisting of 5 sentences. The goal is to predict the sequence of numbers which represent the order of these sentences.
For example, given a story:
He went ...
0
votes
1
answer
2k
views
How do we reduce the output dimensions of BERT?
The output dimensions of BERT are 768-dimensional, is it possible to reduce them to a lower, custom number? For example, if there is another BERT-based transformer model which has a lower count of ...
- 1
0
votes
0
answers
136
views
Why is BERT/GPT capable of "for-all" generalization?
As shown in the figure:
Why does token prediction work when "Socrates" is replaced with "Plato"?
From the point of view of symbolic logic, the above example effectively performs ...
- 235
0
votes
0
answers
217
views
Fine tuning BERT for token level classification
I want to try self-supervised and semi-supervised learning for my task, which relates to token-wise classification for the 2 sequences of sentences (source and translated text). The labels would be ...
5
votes
2
answers
1k
views
What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?
In PyTorch, transformer (BERT) models have an intermediate dense layer in between attention and output layers whereas the BERT and Transformer papers just mention the attention connected directly to ...
1
vote
0
answers
113
views
How to fine-tune a model which was pre-trained on a corpus that contains words with different meanings than the meanings of those words on my corpus?
I have a scenario in which we should leverage previously asked questions (not questions pairs, single question in a column) to locate similar questions within those questions.
How can I fine-tune my ...
- 11
0
votes
1
answer
171
views
Training and Evaluating BERT and XLNET [closed]
I am thinking about a project and have a few questions before I accept it. Would be grateful I anyone experienced of you could give me some advice.
In the project, I have been given a data set with (...
- 19
3
votes
3
answers
6k
views
Isn't attention mask for BERT model useless?
I have just dived into deep learning for NLP, and now I'm learning how the BERT model works. What I found odd is why the BERT model needs to have an attention mask. As clearly shown in this tutorial ...
- 153
2
votes
1
answer
3k
views
Does BERT freeze the entire model body when it does fine-tuning?
Recently, I came across the BERT model. I did some research and tried some implementations.
I wanted to tackle a NER task, so I chose the BertForSequenceClassifications provided by HuggingFace.
...
- 51
2
votes
1
answer
69
views
Why are BERT embeddings interpreted as representations of the corresponding words?
It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a ...
2
votes
0
answers
24
views
Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)
I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:
...
2
votes
0
answers
90
views
What's new in LaBSE v2?
I can't find what's new in LaBSE v2 (https://tfhub.dev/google/LaBSE/2). What are the main highlights of v2 versus v1? And how did you find out?
- 121
1
vote
1
answer
219
views
Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?
Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
- 119
17
votes
2
answers
10k
views
Why does GPT-2 Exclude the Transformer Encoder?
After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
- 321
1
vote
2
answers
1k
views
Should I need to use BERT embeddings while tokenizing using BERT tokenizer?
I am new to BERT and NLP and I am a little confused with tokenization and word embedding.
My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use ...
- 111
1
vote
1
answer
7k
views
How do I calculate the probabilities of the BERT model prediction logits?
I might be getting this completely wrong, but please let me first try to explain what I need, and then what's wrong.
I have a classification task. The training data has 50 different labels. The ...
- 113
2
votes
0
answers
38
views
Adding corpus to BERT for QA
I was wondering about SciBERT's QA abilities using SQuAD. I have a scarce textual dataset consisting of less than 100 files where doctors are discussing cancer in dialogues. I want to add it to ...
- 121
0
votes
1
answer
79
views
Sentiment analysis does not handle neturals [closed]
I'm writing some financial tools, I've found highly performant models for question and answering but when it comes to sentiment analysis I haven't found anything that good. I'm trying to use ...
- 115
1
vote
1
answer
280
views
How to keep track of the subject/entity in a sentence?
I'm working on Sentiment Analysis, using HuggingFace to perform sentiment analysis on articles
...
- 115
1
vote
0
answers
476
views
What does the outputlayer of BERT for masked language modelling look like?
In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows:
In technical terms, the prediction of the output words ...
- 555
2
votes
0
answers
417
views
T5 or BERT for sentence correction/generation task?
I have sentences with some grammatical errors , with no punctuations and digits written in words... something like below:
As you can observe, a proper noun , winston isnt highlighted with capital in ...
- 121
1
vote
1
answer
2k
views
What is MNLI-(m/mm)?
I came across the term MNLI-(m/mm) in Table 1 of the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I know what MNLI stands for, i.e. Multi-Genre Natural ...
2
votes
1
answer
569
views
Transformers: how to get the output (keys and values) of the encoder?
I was reading the paper Attention Is All You Need.
It seems like the last step of the encoder is a LayerNorm(relu(WX + B) + X), i.e. an add + normalization. This should result in a $n$ x $d^{model}$ ...
- 1,510
5
votes
2
answers
678
views
Transformers: how does the decoder final layer output the desired token?
In the paper Attention Is All You Need, this section confuses me:
In our model, we share the same weight matrix between the two embedding layers [in the encoding section] and the pre-softmax linear ...
- 1,510