0
$\begingroup$

I'm new to machine learning and especially, deep learning. Given a video (and it's subtitle), I need to generate a 10-second summary out of this video. How can I use ML and DL to produce the most representative summary out of this video? More specifically, given video scenes, what are some ways to select and rank them, and how to do it? Any ideas would be helpful.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It’s seems like quite challenging problem; at least you would need quite a lot of annotated data and computational power.

The approaches/optimizations you could consider:

  • To make scene change detection and take short piece out of each
  • To introduce some kind of “novelty” metric and try to maximize it to get most different parts of video
  • To convert video to kind of vector with existing solution like r-cnn and yolo and then process it with recurrent networks.
  • The task seems to be very close to video capturing/summarization, you can take inspiration there
  • Also, the attention approach might be handy, look, for example self-attention for video, semantic attention for video
Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks. Yeah first one I'm doing already. Basically all conventional non-DL based approaches. Can you elaborate more on r-cnn and how to use YOLO? The latter is only for object detection, so not sure how. $\endgroup$
    – Mary
    Jul 25, 2019 at 15:24
  • 1
    $\begingroup$ R-cnn solves the same task as yolo, just another approach. The problem with video is that they are very high dimensional and you would need a really big dataset to train directly. For example - Mnist is 28x28 and you need 50’000 to get result - Imagenet 224x224x3 and you need 14’000’000 images - 5 minutes video would be 640x480x3x7200. So you would need billions of annotated videos to train directly. $\endgroup$
    – Kirill Fedyanin
    Jul 26, 2019 at 8:45
  • 1
    $\begingroup$ It’s basically impossible, so you need to reduce the dimensiality in some way. With yolo instead of images you can construst the vector that represents that “we have two objects that look like humans standing near object looking like car”, so, the dimensions would be like 2000x7200 and it would be more managable for DL approaches. Another way would be to understand how you on your own ranking the scenese and why; then you can try manual feature engineering. Here I would argue yolo would be helpful as well, because it’s easier to deal with some semantic data then raw images. $\endgroup$
    – Kirill Fedyanin
    Jul 26, 2019 at 8:45
  • $\begingroup$ I see. One idea came to my mind is to tag those shots I'm interested as 1 and others as 0, and train a model on only those shots against the rest. How do you think it will work? And what's the process? $\endgroup$
    – Mary
    Jul 26, 2019 at 20:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .