Hot to calculate Maximum Normalized log Probability for Active Learning with BERT

I have encountered difficulties understanding the calculation of Maximum Normalized Log Probabilities acording to Shen et al..

With n being the sequence length, yi the label of word i. Xij is the representation (the input).

Let me describe my setting. I'm using the implementation of BERT provided by pytorch to finetune a BERT-Base model for sequence labeling. The goal is to determine a label for each word in the sequence. There are three possible labels for each word. The basic model is set up and runs no problem. Implementing Active Learning based on Maximum Normalized Log Probability is not the problem. Im just not sure if i understand the formular correcly. The Model output is a 128 long (since each sequence is 128 words. Overflow from wordPiece Tokenization is not relevant) list of Labels.

So far i'm using the softmax-function to get the probability for each label instead of the logits the BERT-Model provides as i hand it the unlabeled data. Then i calculate the log probabilities for every label via the log10 function. In the end I sum up the maxima of the predictions over all the words (=the probability of the predicted label for each word) and average them. This is my MNLP value i use to identify the most uncertain instances.

Am i doing it right? My main problem is the max with lowered yi ... yn. As far as i understand it, its used to indicate that only the maximum of the label likelihoods per word is relevant, aka. the probability of the predicted label.

• I am actually working on something similar, BERT-based NER for medical entities. Unfortunately, I too am currently confused on the same bit, on how to incorporate MNLP for AL. Did you have any success in this regard? Aug 11, 2022 at 10:32

It looks like it is the greedy approximation: $$\max_{y_1, ..., y_n} \frac{1}{n} \sum_{i=1} \text{log} P(y_i | y_1, ..., y_{n-1}, x_i) \approx \frac{1}{n} \sum_{i=1} \text{log} \max_{y_i} P(y_i| y_1, ..., y_{n-1}, x_i)$$