1
$\begingroup$

I've been considering starting a project for some time on sound source identification.

To be more specific, my goal is to be able to identify the "sources" for sound in videos. Moving parts clanging, lips speaking, hands clapping etc. I'd like to think a model trained to be able to do this might be helpful for:

  • Identifying who is saying what in a crowd
  • Discovering noise sources caught on video (ex. a carpenter's saw as he is talking to someone)
  • Extending this to design a model for reading lips, to discern speech in silent video.

Helen Keller talking (Taken from https://www.youtube.com/watch?v=8ch_H8pt9M8)

You might think of this task like grounding in NLP, expect for sound/speech instead. I'm sure this has been done before, and I'd like to conduct a literature review. So is there a name for this kind of sound-source identification?

I've tried Googling "Sound Source Identification", but it only returns Speech Classification results (Is this sound a car or a truck etc.)

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ There is "speaker identification", which is your first bullet point, although I don't know about the others. I think the model for reading lips would be separate from the rest of what you are trying to do however, since the rest of it can utilize the sound in the video. $\endgroup$ – Varun Vejalla Aug 9 '20 at 23:32
  • $\begingroup$ Well to my knowledge, "speaker identification" is solely an audio problem: "Is the person currently talking someone I know, and if so who?". $\endgroup$ – madprogramer Aug 10 '20 at 8:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.