From what I have seen you can fine tune a Bert model to detect emotions by labelling single sentences. But if the text you want to evaluate is a large script with many sentences, do I need to split the script into sentences and get a classification for each one? If I did it that way I could find the average score for each emotion for the the sentences in the script. The problem with doing it that way is the the context between sentences is lost. A context like sarcasm often means that sentences have to be connected in order to understand that it is sarcastic. I would be interested to get some advice on this.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
From my understanding of your task, you're looking to get the overall emotion classification score for a long piece of dialogue.
BERT can handle contexts up to 512 tokens in length, so the task becomes a matter of handling differences in training vs testing distributions. If your training set (for recognizing emotions) contains only single sentences, then you're correct in assuming that the model will struggle on longer pieces of text.
The best solution would be to find an in-domain dataset to train on. If you finetune your model on emotion classification for long-contexts, then the model will be able to handle long-contexts natively. In particular, you're looking for conversational datasets.
MELD
One popular dataset is MELD. It consists of dialogue instances from the TV show Friends. Samples have fine-grained labels per utterance, which tend to be pretty short, but they are labeled in-context of the rest of the text. So, you can either train a model to predict the average label of the entire dialogue, or to predict the label of the $i$th utterance in the context of the rest of the dialogue, then averaging the scores after the fact.
This is a pretty popular dataset, so there's a number of existing papers that train models for this task, many of which have code and/or checkpoints available.
EmoryNLP
A similar dataset is the EmoryNLP emotion recognition dataset, part of their character mining project. You can find models trained for this task here.
DailyDialog
Another relevant dataset is the DailyDialog dataset, which features more "every-day" conversational data. You can find models trained for this task here.
Other Datasets
I found many of these (and other) emotion recognition datasets on this paperswithcode listing.
Generating Synthetic Data
Some large language models are trained on enough tasks/data that they may be able to generalize to arbitrary tasks without additional finetuning or with only a few labeled examples. The most famous one would be ChatGPT/GPT-4. You could also try models like FLAN or Tk-Instruct.
Be sure to check the performance of these models manually first, but you can then use these models to label existing dialog data, which you can use to train a smaller model (like BERT). This gives you the flexibility of using any dialog data that's similar to what will be seen during test-time. You can even try prompting GPT-4 to generate similar data to any existing samples that you may already have.
Training your model
I would recommend you take a look at some of these papers to find an architecture/codebase that's effective & easy to replicate, and train on a combination of these datasets. Training on just one may be an issue as these datasets tend to be pretty domain-specific.
I am unsure of how exactly this was trained, but this emotion detection model on huggingface seems to do exactly this. If it was trained on the full dialogs (rather than single sentences/short utterances), then this would work for your use-case -- you may want to test the accuracy of it yourself first.
