I am going to train a deep learning model to classify hand gestures in video. Since the person will be taking up nearly the entire width/height of the video and I will be classifying what hand gesture he or she is doing, I don't need to identify the person and create a bounding box around the person doing the action. I only need to classify video sequences to their class labels.

I will be training on a dataset with individual videos, in which each entire video clip is the particular gesture (So it's a dataset like UCF-101, with video clips corresponding to class labels). But when I am deploying the network, I want the neural network to run on live video. As in how the live video is playing, it should recognize when a gesture has occurred and indicate that it recognized the gesture.

So I was wondering - How can I train the neural network on isolated video sequences in which the entire video clip is the action (like explained above), but run the neural network on live video? For instance, can I use a 3D CNN? Or must I use a 2D CNN with an LSTM network instead, for it to work on live video? My concern is that since a 3D CNN performs the filters across many frames, wouldn't running the CNN on every frame make it very slow? But if I use a 2D CNN with LSTM, will that make it faster? Or will both work fine?

Thank you for your help in advance.


This question actually includes many. I try to answer a couple.

First of all, you need to make sure for your use case to know the non-functional requirements. It is helpful to know that you have a soft real-time case (if harm is done in case of too late predictions it might also be hard real-time)

  1. Latency: how much time may pass after the action was made until the prediction is there?
  2. Stability of prediction: how fast it's the system allowed to change it's prediction? Once every 0.5s?
  3. General hardware, especially memory usage
  4. Optimization metric: maybe accuracy does not tell you enough?
  5. Evaluation setup: likely, you need to have multiple "evaluation points" per video, e.g. Once after the system had one second, once after it had 1.5,... You will also want to consider the stability of the prediction.

Now to the question about how you can do that. If you didn't try it so far, I recommend looking at optical flow.

| improve this answer | |
  • $\begingroup$ Latency and memory consumption are minor problems in Continuous action recognition. It is not a hardware problem which has to do how fast a cpu/gpu is, but a software problem how to implement the Convolutional Neural Network. Under the Wikipedia link to the optical flow nothing is there about isolating video sequences. $\endgroup$ – Manuel Rodriguez Nov 13 '18 at 9:42
  • $\begingroup$ Latency and memory consumption are absolutely critical problems for many computer vision use cases. They can be what makes a complicated technique obviously not worth a try, simply because the technique cannot possibly hold the non-functional requirements.If people ignore that, they can spend weeks or months on an useless approach that could have been discarded with 30min. $\endgroup$ – Martin Thoma Nov 13 '18 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.