Hot to calculate Maximum Normalized log Probability for Active Learning with BERT

Question

I have encountered difficulties understanding the calculation of Maximum Normalized Log Probabilities acording to Shen et al..

With n being the sequence length, yi the label of word i. Xij is the representation (the input).

Let me describe my setting. I'm using the implementation of BERT provided by pytorch to finetune a BERT-Base model for sequence labeling. The goal is to determine a label for each word in the sequence. There are three possible labels for each word. The basic model is set up and runs no problem. Implementing Active Learning based on Maximum Normalized Log Probability is not the problem. Im just not sure if i understand the formular correcly. The Model output is a 128 long (since each sequence is 128 words. Overflow from wordPiece Tokenization is not relevant) list of Labels.

So far i'm using the softmax-function to get the probability for each label instead of the logits the BERT-Model provides as i hand it the unlabeled data. Then i calculate the log probabilities for every label via the log10 function. In the end I sum up the maxima of the predictions over all the words (=the probability of the predicted label for each word) and average them. This is my MNLP value i use to identify the most uncertain instances.

Am i doing it right? My main problem is the max with lowered yi ... yn. As far as i understand it, its used to indicate that only the maximum of the label likelihoods per word is relevant, aka. the probability of the predicted label.

Answer 1

Yes, you are doing it right.

There is an implementation of MNLP here: https://github.com/VectorInstitute/DANER/blob/2b5feadda6c1d35ab7be09277339c490396bc49e/backend/models/al_model.py#L193

It looks like it is the greedy approximation: $\max_{y_1, ..., y_n} \frac{1}{n} \sum_{i=1} \text{log} P(y_i | y_1, ..., y_{n-1}, x_i) \approx \frac{1}{n} \sum_{i=1} \text{log} \max_{y_i} P(y_i| y_1, ..., y_{n-1}, x_i)$

taking maximum log probability of each step individually.

In principle one could use beam search instead of just greedy. However, this only applies to models like CRFs or LSTMs where the labels of each step are not indpendent. For vanilla BERT it doesn't make a difference.