1
$\begingroup$

I was considering a scenario where natural language processing (NLP) and computer vision (CV) are combined, for example in extended reality systems that get as input both natural language and non-verbal information, e.g. human gestures, and can comprehend it. For example, the agent would get language and non-verbal input and talk to a user.

How could this be realized? My naive guess would be a conditional transformer, where the conditioning happens on the non-verbal input, but I'm not sure how exactly the conditioning could happen. What is a current state-of-the-art model for combining NLP and CV?

Also, are there datasets available for the aforementioned use case? I'm thinking of the scenario where a sentence, e.g.

Yeah, I like him too!

can either mean what is said, and the non-verbal input could be that the person saying it is smiling. However, if the non-verbal input is some laughter, then the above sentence might be meant ironically. Is there any dataset for this, where sentences and non-verbal inputs are combined? (Please note that I'm not talking about the generation of a sentence to an image, I'm referring to a combination of NLP and CV.)

Thanks a lot!

Improve this question
$\endgroup$
1
  • $\begingroup$ I don't know the answer, but just in case no one else responds, I can tell you where to look. That's called "multimodal sentiment analysis" in the literature, and the answer you're looking for is probably in this Google Scholar search. $\endgroup$
    – Lee Reeves
    Jun 11, 2022 at 21:04

1 Answer 1

Reset to default
1
$\begingroup$

Your question is very interesting !

Actually your example is about multimodal sarcasm detection, one of the last paper on this task (as far I know) is Detection of Sarcasm through Tone Analysis on video and Audio files: A Comparative Study On Ai Models Performance (The smallest title of the year) They create their own dataset using humoristic Youtube video and labelled it manually. From the title, it seems to be talking about CV but actually, they are using the first 13 dimensions of MFCC thanks to that they get information about prosody, (pitch and tone) and the shape of the vocal tract, the authors add that information to textual data as input to train their models. (LSTM, CNN and Bi-LSTM) Images/videos are not that used yet in Sarcasm identification.

So, nowadays I think the most achieved combination of NLP and CV is the image text extraction task and its language translation (used in Google Lens for example) which is a combination of Optical Character Recognition and Google Neural Machine Translator.

If you are looking for a dataset to join CV and NLP recently there is VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .