All Questions
Tagged with bert machine-learning
8 questions
3
votes
1
answer
833
views
why we use learnable positional encoding instead of Sinusoidal positional encoding
In the original paper of transformers they using positional encoding to capture the position of each word in the sentence and for calculate that it using sin and cos ,like shom in the image. In Bert ...
0
votes
1
answer
2k
views
How do we reduce the output dimensions of BERT?
The output dimensions of BERT are 768-dimensional, is it possible to reduce them to a lower, custom number? For example, if there is another BERT-based transformer model which has a lower count of ...
- 1
0
votes
0
answers
259
views
Fine tuning BERT for token level classification
I want to try self-supervised and semi-supervised learning for my task, which relates to token-wise classification for the 2 sequences of sentences (source and translated text). The labels would be ...
2
votes
1
answer
687
views
Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?
I was wondering if the BERT or T5 models can do the task of generating sentences in English. Most of the models I have mentioned ...
- 263
1
vote
1
answer
336
views
Why does the BERT NSP head linear layer have two outputs?
Here's the code in question.
https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491
...
- 197
3
votes
0
answers
573
views
Can Bert be used to extract embedding for large categorical features?
I've lot of training data points (i.e in millions) and I've around few features but the issue with that is all the features are categorical data with 1 million+ categories in each.
So, I couldn't use ...
- 149
5
votes
1
answer
787
views
Will BERT embedding be always same for a given document when used as a feature extractor
When we use BERT embeddings for a classification task, would we get different embeddings every time we pass the same text through the BERT architecture? If yes, is it the right way to use the ...
11
votes
2
answers
13k
views
What are the segment embeddings and position embeddings in BERT?
They only reference in the paper that the position embeddings are learned, which is different from what was done in ELMo.
ELMo paper - https://arxiv.org/pdf/1802.05365.pdf
BERT paper - https://arxiv....
- 163