How to use pre-trained BERT to extract the vectors from sentences?

Question

I'm trying to extract the vectors from the sentences. Spent soo much time searching for the pre-trained BERT models but found nothing.

Is it possible to get the vectors using pre-trained BERT from the data?

Answer 1

Did you mean:

How do you use a pre-trained BERT model in a feature-based setting to get pre-trained word contextual embeddings?

Here is the BERT paper. I highly recommend you read it.

Firstly, by sentences, we mean a sequence of word embedding representations of the words (or tokens) in the sentence. Word embeddings are the vectors that you mentioned, and so a (usually fixed) sequence of such vectors represent the sentence input. (We don't need the input to always be divided to individual sentences)

There are mainly two ways to train the modern neural network models for sequence modeling tasks such as language modelling, machine translation and question answering:

1. The pretrain - finetune based approach (also called Transfer Learning)

In this approach, a model is first pre-trained on some auxiliary dataset whose domain usually overlaps with the domain of the dataset corresponding to the downstream task, that is, the actual task for which we need to build the model.

In the case of Natural Language Processing (NLP) tasks, such as machine translation and question answering, the pre-training that is done is usually unsupervised (no labels in the dataset) where our training objective is to maximize the log likelihood of the sentences. You can read more about Maximum Likelihood estimation here.

Its almost always a good idea to pre-train models (whether it is for NLP tasks or for computer vision tasks etc). The more we pre-train a model, the closer its parameters are to the optimal ones, and the work needed in the fine-tuning phase is minimized. This is why we use large scale datasets for pre-training.

Later, during the fine-tuning phase, we fine tune the parameters of the model to suit a downstream task, such as a task of translation between two specific languages of text. Since the parameters of the model are already at a position in the error/loss space that is already sort of good enough for the model (because we already pre-trained it with text in, say, 10 languages), the fine-tuning can, in a sense, tailor the model to suit the task a hand.

2. The feature based approach

In this approach, we take an already pre-trained model (any model, e.g. a transformer based neural net such as BERT, which has been pre-trained as described in the previous approach) and then we extract the activations from one or more layers of this pre-trained model.

This is done by simply inputting the word embedding sequence corresponding to a sentence to the pre-trained model and then extracting the activations from one or more layers (one or more of the last layers, since the features associated to the activations in these layers are far more complex and include more context) of this pre-trained model.

These activations (also called contextual embeddings) are used as input (similar to the word embeddings described earlier) to another model, such as an LSTM or even another BERT. That is, fixed features are extracted from the pre-trained model.

Usually the size of these layers are the same as the size of the input layer, i.e. they are also a sequence, but a sequence of far more complex features instead of word embeddings. If the word embedding vector of the input to this model id $D$ and the length of the sentence which is input to the model is $L$, then each of the hidden activation layers have the same length $L$, but the hidden size of these vectors is $H >> D$, i.e. far higher to incorporate more complex features.

Even though the dimensionality of the hidden layers is far higher, these layers are equal in length to the input layer, and so we have a one to one correspondence between the input tokens in the sentence and each of these hidden layer activations.

The bonus in using these hidden representations is that we get to use representations of the input sentence that also incorporate CONTEXT, dependency between the words of the sentence, an important feature that is to be modeled by all models used for NLP tasks. Models such as BERT, are based on Transformer models that use an attention mechanism in order to model these contextual features.

This is a superb article describing to the last detail the attention mechanism and the Transformer model based on which models like BERT operate. For more about contextual embeddings I recommend taking a look at this article. The figure used here is from this article.

Models such as BERT make use of one half of these Transformer models, called the encoder, since the Transformer is a seq2seq or encoder-decoder model. Here is the link to the Transformer paper.

Here is a great blog on extracting contextual word embeddings from BERT using Tensorflow and Keras.

I'll also provide a link to a Kaggle Python Notebook on using Pipelines functionality from the HuggingFace community repo on github that also is used for feature extraction (contextual embeddings).

Please let me know if you have any questions, happy to help!