How to train a sequence labeling model with annotations from three annotators?

Question

I have a dataset of movie reviews annotated by 3 persons. The following example contains one sentence with corresponding annotations from 3 different persons.

sentence = ['I', 'like', 'action', 'movies','!']
annotator_1 = ['O','O', 'B_A', 'I_A', 'O'] 
annotator_2 = ['O','O', 'B_A', 'I_A', 'O'] 
annotator_3 = ['O','O', 'B_A', 'O', 'O']

The labels follow the BIO format. That is, B_A means the beginning of aspect-term (action) and I_A indicates inside of aspect-term (movie). Unfortunately, the annotators do not agree always together. While the first two persons assigned the right labels for aspect-term (action movie), the last one mislabeled the token (movies).

I am using Bi-LSTM-CRF sequence tagger to train the model. However, I am not sure if am using the training data correctly.

Is it correct to feed the model the same sentence with annotations from 3 persons? Then test it in the same way, i.e., the same sentence with different annotations?

Another question.

I merged the annotations in one final list of labels as follows:

final_annotation = ['O','O', 'B_A', 'I_A', 'O']

In this case, the final label is chosen based on the majority of labels among three annotators.

Is it right to feed the model the same sentence with corresponding annotations from all users during the testing phase?

Answer 1

Both ways are valid. It depends on what you want from the model and expect from the data. Generally though I would use 1 assumption and stick with it (unless there was a specific reason not to), so I would use all lines for test if training done that way, and same for majority.

Also note if you ever get more than 3 people, you can choose to do a variance based approach (use data if only x% agree, throw away otherwise (or you could even weigh controversial labels lower))