Questions tagged [bert]

For questions related to BERT (which stands for Bidirectional Encoder Representations from Transformers), a language representation model introduced in the paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (2019) by Google.

Filter by
Sorted by
Tagged with
0
votes
0answers
10 views

What's new in LaBSE v2?

I can't find what's new in LaBSE v2 (https://tfhub.dev/google/LaBSE/2). What are the main highlights of v2 versus v1? And how did you find out?
0
votes
0answers
8 views

What is the recommended way to retokenize ontonotes to wordpiece?

I am using ontonotes5 dataset which is already tokenize. But since I want to use Bert. I want to tokenize it myself, keeping all the annotations correct.
0
votes
0answers
19 views

Cannot add CRF layer on top of BERT in keras for NER [closed]

I am facing an unknown issue while training my BERT-CRF model for NER. I am using keras.contrib for the CRF model. Here are the imported libraries. ...
0
votes
1answer
34 views

Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?

Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
1
vote
1answer
43 views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
0
votes
0answers
13 views

Adding BERT embeddings in BiLSTM embedding layer

I am want to use BERT embeddings in the BiLSTM embedding layer instead of Word2Vec or FastText Embeddings. There is any code to do that?
0
votes
1answer
27 views

Should I need to use BERT embeddings while tokenizing using BERT tokenizer?

I am new to BERT and NLP and I am a little confused with tokenization and word embedding. My doubt is if I use the BertTokenizer for tokenizing a sentence then do I have to compulsorily use ...
0
votes
0answers
17 views

How does BERT answer questions?

I have been trying to understand how the BERT model works. Specifically, I am trying to understand how it picks up answers to questions on a given passage. I have tried following this blog post and, ...
1
vote
1answer
106 views

How do I calculate the probabilities of the BERT model prediction logits?

I might be getting this completely wrong, but please let me first try to explain what I need, and then what's wrong. I have a classification task. The training data has 50 different labels. The ...
0
votes
0answers
10 views

Understanding Huggingface QA Transformers Confidence Scores

I seem to be unable to find information on whether the confidence scores for the huggingface transformers used for QA are in the form of 10^-2 or 10^-5. What are your thoughts on the matter? Example: <...
2
votes
0answers
20 views

Adding corpus to BERT for QA

I was wondering about SciBERT's QA abilities using SQuAD. I have a scarce textual dataset consisting of less than 100 files where doctors are discussing cancer in dialogues. I want to add it to ...
0
votes
1answer
27 views

Sentiment analysis does not handle neturals [closed]

I'm writing some financial tools, I've found highly performant models for question and answering but when it comes to sentiment analysis I haven't found anything that good. I'm trying to use ...
1
vote
1answer
31 views

How to keep track of the subject/entity in a sentence?

I'm working on Sentiment Analysis, using HuggingFace to perform sentiment analysis on articles ...
1
vote
0answers
27 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
0
votes
0answers
16 views

How to chose a specific contextual embedding?

I am learning deep learning using a plethora of online resources. I have been using mostly word2vec and gloVe for my NMT model. I recently came across the concept of contextual-embedding, and I see ...
0
votes
0answers
15 views

Is parameter sharing in AlBERT akin to repeated application of same function on input?

I read the AlBERT and saw that its architecture used "Parameter Sharing" among layers of the encoder. They mentioned that this was done to save model space, make fewer training parameters ...
2
votes
0answers
52 views

T5 or BERT for sentence correction/generation task?

I have sentences with some grammatical errors , with no punctuations and digits written in words... something like below: As you can observe, a proper noun , winston isnt highlighted with capital in ...
1
vote
0answers
106 views

What is MNLI-(m/mm)?

I came across the term MNLI-(m/mm) in Table 1 of the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I know what MNLI stands for, i.e. Multi-Genre Natural ...
2
votes
1answer
40 views

Transformers: how to get the output (keys and values) of the encoder?

I was reading the paper Attention Is All You Need. It seems like the last step of the encoder is a LayerNorm(relu(WX + B) + X), i.e. an add + normalization. This should result in a $n$ x $d^{model}$ ...
2
votes
1answer
31 views

Transformers: how does the decoder final layer output the desired token?

In the paper Attention Is All You Need, this section confuses me: In our model, we share the same weight matrix between the two embedding layers [in the encoding section] and the pre-softmax linear ...
0
votes
0answers
20 views

Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?

I know that several tokenization methods that are used for tranformer models like WordPiece for Bert and BPE for Roberta and others. What I was wondering if there is also a transformer which uses a ...
1
vote
1answer
102 views

Is there a way to provide multiple masks to BERT in MLM task?

I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence. ...
0
votes
0answers
18 views

What BERT predicts when the token supposed to be masked is not masked?

I am reading the BERT paper. In the paper, they say that: Although this allows us to obtain a bidirec- tional pre-trained model, a downside is that we are creating a mismatch between pre-training and ...
1
vote
0answers
200 views

Why aren't the BERT layers frozen during fine-tuning tasks?

During transfer learning in computer vision, I've seen that the layers of the base model are frozen if the images aren't too different from the model on which the base model is trained on. However, on ...
4
votes
0answers
45 views

Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?

BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
0
votes
1answer
40 views

BERT: After pretraining 880000 step, why fine-tune not work? [closed]

I am using pretraining code from https://github.com/NVIDIA/DeepLearningExamples Pretrain parameters: ...
2
votes
0answers
38 views

Bert for Sentiment Analysis - Connecting final output back to the input

I have not found a lot of information on this, but I am wondering if there is a standard way to apply the outputs of a Bert model being used for sentiment analysis, and connect them back to the ...
0
votes
0answers
16 views

How to use an image tensor for caption generation with Transformer-XL or BERT?

I am fairly new to transformers and deep learning in general so please be kind, I am currently working on a project that will caption images using either Transformer-XL or BERT, however, I am not sure ...
2
votes
1answer
709 views

How is BERT different from the original transformer architecture?

As far as I can tell, BERT is a type of Transformer architecture. What I do not understand is: How is Bert different from the original transformer architecture? What tasks are better suited for BERT,...
0
votes
0answers
22 views

Why BERT last 4 Layers should be considered to extract word embeddings?

In most of the cases the embedding vectors of last 4 layers of BERT are summed up to represent the tokens embedding. I've tried to explore but haven't found any strong reason/resource on why we should ...
0
votes
0answers
12 views

What is the best NLP model for handeling a rapid extension of the production data for a question answeing task

I'm lately writing a chat-bot to answer questions about children's fantasy books I save to my database . When a user opens a book it loads the chat-bot for the given book . I don't need it to work on ...
0
votes
0answers
23 views

Automated Scoring (non-english language) Using BERT

i'm a student and i'm new to NLP. I want to build an Automated Scoring system which is in Indonesian Language using BERT. The system is expected to be able to measure the similarity of an answer(e.g: ...
0
votes
0answers
25 views

Comparing English and Spanish Queries using NLP

I am trying to see whether there are differences in sentences between the topics in 2 languages ie English and Spanish. Eg. (Face masks are mandatory, la mascarilla es obligatoria) but this stretched ...
4
votes
1answer
2k views

How to fine tune BERT for question answering?

I wish to train two domain-specific models: Domain 1: Constitution and related Legal Documents Domain 2: Technical and related documents. For Domain 1, I've access to a text-corpus with texts from ...
1
vote
1answer
123 views

How to use speaker's information as well as text for fine-tuning BERT?

I want to classify my corporate chat messages into a few categories such as question, answer, and report. I used a fine-tuned BERT model, and the result wasn't bad. Now, I started thinking about ways ...
1
vote
1answer
159 views

Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?

I was wondering if the BART or T5 models can do the task of generating sentences in English. Most of the models I have mentioned ...
0
votes
1answer
60 views

Why does the BERT NSP head linear layer have two outputs?

Here's the code in question. https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491 ...
1
vote
0answers
30 views

Two questions about the architecture of Google Bert model (in particular about parameters)

I'm looking for someone who can help me clarify a few details regarding the architecture of Bert model. Those details are necessary for me to come with a full understanding of Bert model, so your help ...
1
vote
0answers
23 views

Is the number of bidirectional LSTMs in seq2seq model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
0
votes
1answer
48 views

How to add a pretrained model to my layers to get embeddings?

I want to use a pretrained model found in [BERT Embeddings] https://github.com/UKPLab/sentence-transformers and I want to add a layer to get the sentence embeddings from the model and pass on to the ...
2
votes
1answer
178 views

Similarity score between 2 words using Pre-trained BERT using Pytorch

I'm trying to compare Glove, Fasttext, Bert on the basis of similarity between 2 words using Pre-trained Models. Glove and Fasttext had pre-trained models that could easily be used with gensim ...
1
vote
1answer
2k views

How to use pre-trained BERT to extract the vectors from sentences?

I'm trying to extract the vectors from the sentences. Spent soo much time searching for the pre-trained BERT models but found nothing. Is it possible to get the vectors using pre-trained BERT from ...
7
votes
1answer
568 views

What is the intuition behind the dot product attention?

I am watching Attention all you need, In that what is the intuition behind the dot product attention? $$A(q,K, V) = \sum_i\frac{e^{q.k_i}}{\sum_j e^{q.k_j}} v_i$$ becomes: $$A(Q,K, V) = softmax(QK^...
2
votes
0answers
23 views

Will structured knowledge bases continue to be used in question answering with the likes of BERT gaining popularity?

This may come across as an open and opinion-based question, I definitely want to hear expert opinions on the subject, but I am also looking for references to materials that I can read deeply. One of ...
1
vote
0answers
75 views

Building a template based NLG system to generate a report from data

I am a newbie to NLP and NLG. I am tasked to develop a system to generate a report based on a given data table. The structure of the report and the flow is predefined. I have researched on several ...
1
vote
0answers
134 views

Continuing Pretraining on BERT

I need some help with continuing pre-training on Bert. I have a very specific vocabulary and lots of specific abbreviations at hand. I want to do a STS task. Let me specify my task: I have ...
1
vote
0answers
35 views

Tensorflow based implementation of Text classification with any variation of BERT(ALBERT/XLNET)

Do you have any reference you can point me to for doing Text classification using any variation of BERT(albert or XLnet) with a TF implementation. I am not sure how to deploy torch based models, so ...
0
votes
2answers
439 views

Why is my loss (binary cross entropy) converging on ~0.6? (Task: Natural Language Inference)

I’m trying to debug my neural network (BERT fine-tuning) trained for natural language inference with binary classification of either entailment or contradiction. I've trained it for 80 epochs and its ...
3
votes
0answers
231 views

Can Bert be used to extract embedding for large categorical features?

I've lot of training data points (i.e in millions) and I've around few features but the issue with that is all the features are categorical data with 1 million+ categories in each. So, I couldn't use ...
6
votes
1answer
836 views

How to use BERT as a multi-purpose conversational AI?

I'm looking to make an NLP model that can achieve a dual purpose. One purpose is that it can hold interesting conversations (conversational AI), and another being that it can do intent classification ...