I'd like to generate subtitles for a silent film. Is there an open source project out there capable of creating captions based on a series of images (such as a scene from a movie)?

EDIT: thanks for the comments below. To clarify, what i'm looking for is an algorithm which can generate a caption for a sequences of images within a movie describing what happens in the sequence. This is for preliminary research, so accuracy is less important.

  • $\begingroup$ Welcome to ai...I suggest you specify you want the captions based on what more specifically with some examples.. Since it is a silent film it's not based on audio. $\endgroup$
    – user9947
    Feb 28, 2018 at 13:02
  • $\begingroup$ This is a really cool question. My guess, based on current limitations, is that there is no AI in existence that could come close to providing this function... today. But even as AI advances, this particular function will be challenging. The AI has to understand the context AND subtext of the silent scenes. (Many humans seem not to get subtext;) Both are far beyond current capabilities, but I'm sure facial recognition related to emotions will be a major part of the puzzle. $\endgroup$
    – DukeZhou
    Feb 28, 2018 at 18:56
  • 1
    $\begingroup$ What does a subtitle mean for a silent film? Silent films don't have spoken dialogue. If the actors are not speaking what is there to put in a subtitle? Are you asking how to summarize the actions of video segments into words? If that is the case you may find the following video of interest although it does not directly address your question: youtube.com/watch?time_continue=1&v=AR3hY9iB5-I $\endgroup$ Mar 1, 2018 at 4:07
  • $\begingroup$ Check out Amazon's Rekognition which combines algorithms like object detection, tracking,facial recognition, activity detection, and celebrity recognition: aws.amazon.com/rekognition and youtube.com/watch?v=SNONL4IecHE $\endgroup$ Mar 1, 2018 at 4:26
  • $\begingroup$ Added a clarification $\endgroup$ Mar 1, 2018 at 22:17

3 Answers 3


What you might want to look for is called video captioning. Earlier examples from this line of research is:

Below is a screenshot of the results (positive and negative) reported in those papers: Example from Venugopalan1 Example from Pasunuru

For the ICCV paper, it's not hard to find some implementations, e.g. here.

For more recent results, I would suggest to look into the ActivityNet 2017 Challenge - dense captioning or its 2018 version. Some winning solutions include:

However I am not sure whether any open-source implementation has been released.


Here is one open-source implementation. Temporal Tessellation: A Unified Approach for Video Analysis

For more you can dig into some research publications and see if they give a link to their implementation. Most researchers put up their work for public.

Here is a list of publications that present their work related to video captioning using machine learning. Awesome Deep Vision

Here is another publication that shows how to generate captions for videos. Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks


No , even generating brief summary of a video is out of scope as of current state of the art. Training such a model is a tough task. Video understanding is pretty far from the research as of now. But you could try generating descriptions of some keyframes of the video , and align them to form meaningful passages.

check this https://arxiv.org/abs/1611.06607


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .