I have a dataset of movie reviews annotated by 3 persons. The following example contains one sentence with corresponding annotations from 3 different persons.
sentence = ['I', 'like', 'action', 'movies','!']
annotator_1 = ['O','O', 'B_A', 'I_A', 'O']
annotator_2 = ['O','O', 'B_A', 'I_A', 'O']
annotator_3 = ['O','O', 'B_A', 'O', 'O']
The labels follow the BIO format. That is, B_A
means the beginning of aspect-term (action) and I_A
indicates inside of aspect-term (movie).
Unfortunately, the annotators do not agree always together. While the first two persons assigned the right labels for aspect-term (action movie), the last one mislabeled the token (movies).
I am using Bi-LSTM-CRF sequence tagger to train the model. However, I am not sure if am using the training data correctly.
Is it correct to feed the model the same sentence with annotations from 3 persons? Then test it in the same way, i.e., the same sentence with different annotations?
Another question.
I merged the annotations in one final list of labels as follows:
final_annotation = ['O','O', 'B_A', 'I_A', 'O']
In this case, the final label is chosen based on the majority of labels among three annotators.
Is it right to feed the model the same sentence with corresponding annotations from all users during the testing phase?