# Tag Info

27

For newbies, NO. Sentence generation requires sampling from a language model, which gives the probability distribution of the next word given previous contexts. But BERT can't do this due to its bidirectional nature. For advanced researchers, YES. You can start with a sentence of all [MASK] tokens, and generate words one by one in arbitrary order (instead ...

10

Let's start with a bit of notation and a couple of important clarification. Q refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word. V refers to the values vectors matrix, $v_i$ being a single value vector associated with a single input word. K refers to the keys vectors matrix, $k_i$ being a single key ...

7

What is a transformer? The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-called attention (i.e. a mechanism that determines the importance of words to other words in a sentence or which words are more likely to come together) and the ...

5

this experiment by Stephen Mayhew suggests that BERT is lousy at sequential text generation: http://mayhewsw.github.io/2019/01/16/can-bert-generate-text/ although he had already eaten a large meal, he was still very hungry As before, I masked “hungry” to see what BERT would predict. If it could predict it correctly without any right context, we might be in ...

4

Sentences (for those tasks such as NLI which take two sentences as input) are differentiated in two ways in BERT: First, a [SEP] token is put between them Second, a learned embedding $E_A$ is concatenated to every token of the first sentence, and another learned vector $E_B$ to every token of the second one That is, there are just two possible "...

4

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the MASKING in the multi-head attention block, the decoder is only allowed to glean information ...

3

I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py). It seems like an extra linear layer learns the weights $W^{key}$ and $W^{value}$ (plus biases), so to get the output (keys and values), you multiply the output of the encoder's final add + norm layer by \$...

3

Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead. # Sample code # Model architecture # Custom BERT layer bert_output = BertLayer(n_fine_tune_layers=10)(bert_inputs) # Build the rest of the classifier dense = tf.keras.layers.Dense(256, activation='relu')(bert_output) pred = tf.keras.layers....

3

These embeddings are nothing more than token embeddings. You just randomly initialize them, then use gradient descent to train them, just like what you do with token embeddings.

2

No. Sentence generating is directly related to language modelling (given the previous words in the sentence, what is the next word). Because of bi-directionality of BERT, BERT cannot be used as a language model. If it cannot be used as language model, I don't see how you can generate a sentence using BERT.

2

Think of BERT (or similar models) as as good starting place for understanding context. A couple options to make BERT contextualize dialogue: Concatenate all messages with a seperator embedding and finetune a language model like BERT This has shown good results in this paper, but understand it has weaknesses like struggling to determine order or author ...

2

Did you mean: How do you use a pre-trained BERT model in a feature-based setting to get pre-trained word contextual embeddings? Here is the BERT paper. I highly recommend you read it. Firstly, by sentences, we mean a sequence of word embedding representations of the words (or tokens) in the sentence. Word embeddings are the vectors that you mentioned,...

2

My answer assumes your fine-tuning architecture simply stacks a single fully-connected layer on top of the BERT [CLS] output, as in Figure 4b of the BERT paper. Generally, when working with mixed data such as continuous and categorical features, the first step is to simply concatenate all the inputs into one long vector. In your case, you would concatenate a ...

2

What you're describing is known as coreference resolution. More specifically, this example is anaphora resolution. The short answer is that this is an open research question and there is no well-established solution. You mentioned Hugging Face in your question. The neuralcoref module in spaCy is itself from Hugging Face (note the reflexive anaphor used for ...

1

The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot taking a dialogue context and producing a response. In these cases, there are qualitative differences between the inputs and outputs so that it makes sense to ...

1

Your call to model.predict() is returning the logits for softmax. This is useful for training purposes. To get probabilties, you need to apply softmax on the logits. import torch.nn.functional as F logits = model.predict() probabilities = F.softmax(logits, dim=-1) Now you can apply your threshold same as for the Keras model.

1

Yes, there is. You can try Spacy. Here you go. import spacy from spacytextblob.spacytextblob import SpacyTextBlob nlp = spacy.load('en_core_web_sm') spacy_text_blob = SpacyTextBlob() nlp.add_pipe(spacy_text_blob) text = "i'm good" doc = nlp(text) print(doc._.sentiment.polarity) # 0.7 text = "i'm bad" doc = nlp(text) print(doc._....

1

I found the answer by reading the paper referenced by that section, Using the output embedding to improve language models Based on this observation, we propose threeway weight tying (TWWT), where the input embedding of the decoder, the output embedding of the decoder and the input embedding of the encoder are all tied. The single source/target vocabulary of ...

1

There is a pre-trained language model called ProphetNet for sequence-to-sequence learning with a novel self-supervised objective called future n-gram prediction. https://github.com/microsoft/ProphetNet Also, there are few variants on hugging face website as well https://huggingface.co/models?search=ProphetNet

1

The answer is yes but 'lightweight' will require a 'lightweight' model. Your application for 'domain one' is called open domain question answering (ODQA). Here is a demonstration of ODQA using BERT: https://www.pragnakalp.com/demos/BERT-NLP-QnA-Demo/ Your application for 'domain two' is a little different. It is about learning sequences from sequences. ...

1

This seems to be inherited from the original Google implementation, which also uses 2 outputs (https://github.com/google-research/bert/blob/master/run_pretraining.py#L293). I can see two possible reasons that the original implementation uses 2 outputs: They are using the cross entropy loss, which typically works with log probabilities. To get probabilities ...

1

I think you should use Keras embedding layer. It will be too easier than what you are doing. Steps Create Embedding Matrix add matrix to embedding layer while building model. You will find detailed article https://www.cs.uaf.edu/2011/spring/cs641/lecture/04_05_modeling.html

1

It seems to be overfitting and your model is not learning. Try SGD optimizer with a learning rate of 0.001 ADAM optimizer will give you a soon overfitting, and decreasing the learning rate will train your model better. The learning rate is about steps to change weights, in this plot you see that the validation loss is not changing with an optimization goal

1

Have a look at https://medium.com/the-artificial-impostor/news-topic-similarity-measure-using-pretrained-bert-model-1dbfe6a66f1d You can have the two sentences as first and second use the next sentence score as a similarity measure. You can further fine-tune your model on some semantic similarity tasks like Sent-Eval or your own dataset if you have one

1

BERT is deterministic. There is no variation unless you parse your tokens differently in succeeding runs. Here is the original paper the model architecture is based off of Transformer Paper. Note that in every layer, the only operations used for the most part are matrix multiplications, concatenations, basic ops, and layer normalizations, all of which are ...

1

These answers are based on my personal understanding of Bert from both the paper and official_implementation, hope it will help: What do they mean by "maximum scoring span is used as the prediction"? As you know in SQuAD the input sequence is divided to 2 parts: Question and Document (from which we extract the answer if possible). Sometimes the input ...

1

Maybe the following article can help you: FAQ Retrieval using Query-Question Similarity and BERT-Based Query-Answer Relevance (2019) They evaluate their model in localgovFAQ and StackExchange datasets.

1

What alternate options exist for this? No. Sentence generating is directly related to language modelling (given the previous words in the sentence, what is the next word). Because of bi-directionality of BERT, BERT cannot be used as a language model. If it cannot be used as language model, I don't see how you can generate a sentence using BERT.

1

Of course now there has been a huge development: Huggingface published pytorch-transformers, a library for the so successful Transformer models (BERT and its variants, GPT-2, XLNet, etc.), including many pretrained (mostly English or multilingual) models (docs here). It also includes one German BERT model. SpaCy offers a convenient wrapper (blog post). ...

Only top voted, non community-wiki answers of a minimum length are eligible