I have a dataset of movie reviews annotated by 3 persons. The following example contains one sentence with corresponding annotations from 3 different persons.

sentence = ['I', 'like', 'action', 'movies','!']
annotator_1 = ['O','O', 'B_A', 'I_A', 'O'] 
annotator_2 = ['O','O', 'B_A', 'I_A', 'O'] 
annotator_3 = ['O','O', 'B_A', 'O', 'O']

The labels follow the BIO format. That is, B_A means the beginning of aspect-term (action) and I_A indicates inside of aspect-term (movie). Unfortunately, the annotators do not agree always together. While the first two persons assigned the right labels for aspect-term (action movie), the last one mislabeled the token (movies).

I am using Bi-LSTM-CRF sequence tagger to train the model. However, I am not sure if am using the training data correctly.

Is it correct to feed the model the same sentence with annotations from 3 persons? Then test it in the same way, i.e., the same sentence with different annotations?

Another question.

I merged the annotations in one final list of labels as follows:

final_annotation = ['O','O', 'B_A', 'I_A', 'O']

In this case, the final label is chosen based on the majority of labels among three annotators.

Is it right to feed the model the same sentence with corresponding annotations from all users during the testing phase?


1 Answer 1


Both ways are valid. It depends on what you want from the model and expect from the data. Generally though I would use 1 assumption and stick with it (unless there was a specific reason not to), so I would use all lines for test if training done that way, and same for majority.

Also note if you ever get more than 3 people, you can choose to do a variance based approach (use data if only x% agree, throw away otherwise (or you could even weigh controversial labels lower))

  • $\begingroup$ In the first assumption, sentences are fed into the classification model 3 times with different annotations. Could that lead to overfitting? $\endgroup$
    – Ali F
    Mar 21, 2021 at 17:15
  • $\begingroup$ giving same input with different outputs does not cause overfitting, it creates an internal weighting of how much to trust each value. Depending on data this can also confuse the model. $\endgroup$
    – mshlis
    Mar 22, 2021 at 3:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .