I've been considering starting a project for some time on sound source identification.
To be more specific, my goal is to be able to identify the "sources" for sound in videos. Moving parts clanging, lips speaking, hands clapping etc. I'd like to think a model trained to be able to do this might be helpful for:
- Identifying who is saying what in a crowd
- Discovering noise sources caught on video (ex. a carpenter's saw as he is talking to someone)
- Extending this to design a model for reading lips, to discern speech in silent video.
(Taken from https://www.youtube.com/watch?v=8ch_H8pt9M8)
You might think of this task like grounding in NLP, expect for sound/speech instead. I'm sure this has been done before, and I'd like to conduct a literature review. So is there a name for this kind of sound-source identification?
I've tried Googling "Sound Source Identification", but it only returns Speech Classification results (Is this sound a car or a truck etc.)