I'm new to machine learning and especially, deep learning. Given a video (and it's subtitle), I need to generate a 10-second summary out of this video. How can I use ML and DL to produce the most representative summary out of this video? More specifically, given video scenes, what are some ways to select and rank them, and how to do it? Any ideas would be helpful.
It’s seems like quite challenging problem; at least you would need quite a lot of annotated data and computational power.
The approaches/optimizations you could consider:
- To make scene change detection and take short piece out of each
- To introduce some kind of “novelty” metric and try to maximize it to get most different parts of video
- To convert video to kind of vector with existing solution like r-cnn and yolo and then process it with recurrent networks.
- The task seems to be very close to video capturing/summarization, you can take inspiration there
- Also, the attention approach might be handy, look, for example self-attention for video, semantic attention for video