I know that it is very common for machine learning systems to classify objects based on their visual features such as shapes, colours, curvatures, width-to-length ratios, etc.

What I'd like to know is this: Do any techniques exist for machine learning systems to classify objects based on how they move?


  • Suppose that in a still image, 2 different classes of objects look identical. However, in a video Class A glides smoothly across the screen while Class B meanders chaoticaly across the screen.
  • When given multiple videos of the same person walking, classify whether he's sober or drunk.

1 Answer 1


Machine learning systems work based on input data. The form of that data is irrelevant. It may need some configuring on the technical side of things, but the general ability to learn from the dataset remains the same.
ML systems are built to learn how to interpret data, without you needing to predefine exactly what it should be looking for.

But even then, you can reason that videos are no different from images. A video is essentially nothing but a flipbook of images. If you were to paste every frame of the video in a long sequence, you would have one big picture.
The rest should follow as usual: if a machine learning system can be taught to interpret images, it can evidently also interpret these "frame sequence" images, which means it's able to interpret the dataset that we colloquially call a video.

That being said, since a video contains much more information (a 5 second 30fps video is 150x as much data as a single image with the same resolution), the required learning process increases exponentially.

So yes, it's perfectly possible, but it will require more processing power and training as there are more input variables to account for.

There are ways to reduce this increase in complexity. For example, to check if someone is drunk or not, rather than push the entire video through the learning algorithm, you instead preprocess the video to figure out the person's gait or even just their footsteps (relative location and timing) and only push this processed data through.

This dramatically cuts down on the complexity of the network needed; at the cost of requiring preprocessing which may introduce issues (bad preprocessing = bad learning).


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .