For example, I would like to train my neural network to recognize the type of actions (e.g. in commercial movies or some real-life videos), so I can "ask" my network in which video or movie (and at what frames) somebody was driving a car, kissing, eating, was scared or was talking over the phone.
What are the current successful approaches to that type of problem?
There are several approaches as to how this can be achieved.
One recent study from 2015 about Action Recognition in Realistic Sports VideosPDF uses the action recognition framework based on the three main steps of feature extraction (shape, post or contextual information), dictionary learning to represent a video, and classification (BoW framework).
A few examples of methods:
Spatio-Temporal Structures of Human Poses
a joint shape-motion
Multi-Task Sparse Learning (MTSL)
Hierarchical Space-Time Segments
Spatio-Temporal Deformable Part Models (SDPM)
Here are the results based on training of 10 action classes based on the UCF sports dataset:
This study from 2012 uses 3D convolutional neural networks (CNN) for automated recognition of human actions in surveillance videos. The 3D CNN model extracts features from both the spatial and the temporal dimensions by performing 3D convolutions, thereby capturing the motion information encoded in multiple adjacent frames. A very similar deep learning approach based on 3D CNN is demonstrated in the LIRIS and Orange Labs study from 2011.
This Oxford study from 2014 also uses a similar approach, but with two-stream CNN which incorporates spatial and temporal networks which can achieve good performance despite having limited training data. It recognises action from motion in the form of dense optical flow. For example:
Another study from 2007 demonstrates a method by detecting human falls based on a combination of motion history and human shape variation by analysing the video frames. It uses Motion History Image (MHI) to quantify the motion of the person.
Source: harishrithish7/Fall-Detection at GitHub
An alternative general approach could be action detection based on the posture using DNN. See: How to achieve recognition of postures and gestures?
MIT have done research and implemented an incomplete version of action video recognition.
With the use of MATLAB, NNetworks and a large set of training videos.
My suggested set of comments on my previous answer indicate the usage of a multi interconnected NNet, verus MIT's image based NNet.
A neural network can be used but must be trained to expect the information (pattern of data, pixels or groupings of loose range such as color, and location) at any given location in the network, first a vision system must but implemented. Then a facial recognition, multiple partial individual body fixing (finding body part and there partners to a person) then training on some states and you'll have it work. MIT have done research and have made a seemy accurate implementation.
I'm an AI Researcher and Software Engineer for the past 7 years.
No General Movie Search Yet
There have been successes in recognizing a very narrow sequence of a very narrow set of possible actions, but nothing like a general movie searching system that can return a set of matches with the start time, end time, and movie instance for each match to one of the search criteria listed in this question.
Normalizing the List
First of all, "Was scared," is not the description of an action. It should be, "Becoming scared." Secondly, "Talking over the phone," is not a proper action description. It should be a conjunctive action such as, "Talking into a phone AND listening to the same phone." To make the list homogenous in format, the first item should be "Car driving," since the actor is human in every other case.
Realistic System Design Expectations
It is unrealistic to think that an artificial neural net, by itself, can be trained to return as output the set of start and stop ranges and associated movie instances from a database of movies and one of the above list items as input. This will require a complex system with many ANNs and other ML devices and may require other AI components that are not activation type networks at all. Certainly convolution kernels and various types of encoders should be considered as key system components.
You will need a large amount of training data to cover the above six cases (the last of the five items actually being two distinct actions that we normally associate and consider one). If you want to detect more actions, you will need a large amount of training data for them too.
Verbs and Nouns
The reason this question is interesting to me is that recognizing ACTIONS are not the same as recognizing ITEMS. All mammals learn ITEMS first and ACTIONS later. Linguistically, nouns come before verbs in child language development. That is because, just as detecting edges is preliminary to detecting shapes, which is preliminary to detecting objects, detecting motion is preliminary to detecting action.
Verbs like, "Eating," are an abstraction over the top of the motion, and, in the case of eating, the motion is complex. Also, eating is not the same thing as gum chewing, so the sequence detected must be as follows:
The probability of a sequence is the product of the probability of its parts so that math is simple and easy to implement. Concurrency, as in the case of conjunctive actions like talking into and listening to the same phone, is also relatively easy to handle in general.
A Realistic Approach
Certainly, generalization (and more specifically feature extraction) will need to occur in object recognition, collision detection, motion detection, facial recognition, and other planes simultaneously. A complex topology, perhaps employing equilibria as in GAN design, will most likely be necessary to assemble elements of criteria associated with the movie query string and to run windows over the frames of each movie.
To provide a service that returns results within a few days or weeks will probably require a cluster and DSP hardware (perhaps leveraging GPUs).
Special Cases that Human Brains Handle
Determining how long one of the two elements of concurrency can be undetected before it invalidates the conjunction can be tricky. (How long can one not speak into the phone before it appears that it is no longer considered phone conversation?)
If in the movie, only the swallowing is shown, a human can infer eating. That kind of conclusion reliability from sparse data is a huge AI challenge discussed in various contexts throughout the literature.
The Emergence of Associated Technology — A Projection
I suspect that the system topography comprised of ANNs, encoders, convolution kernels, and other components to perform the search for any of a select set of actions will emerge within the next ten years. Work seems to be tracking in that direction in the literature.
A system that will acquire its own training information, sustainably grow in knowledge and perform general searches if increasing breadth and complexity may be anywhere from forty to two hundred years out. It is difficult to predict.
Gross Overoptimistic Predictions
Every generation seems to view knowledge growth as an exponential function and tends to make unrealistic predictions about the advent of certain coveted technology capabilities. Most of the predictions fail dramatically. I have come to believe that exponential growth is an illusion created by the inverse exponential decay of interest in the past with respect to time.
We lose track of the energy and rate of growth in eras before us because they become socially irrelevant. People into scientific history, like Whitehead, Kuhn, and Ellul know that technology has moved forward quickly for at least a few hundred years. Vernadsky inferred in his The Biosphere that life may not have arisen, that like matter and energy, it may always have existed. I wonder if technology has been moving at an essentially constant rate for the last 50,000 years.
Germany decided to double its solar panel energy output every year and published its exponential success, until a few years ago when doubling it again would cost a hundred billion dollars more than what they had to spend. They stopped publishing the exponential growth graphs.
