How should continuous action/gesture recognition be performed differently than isolated action recognition

Question

I am going to train a deep learning model to classify hand gestures in video. Since the person will be taking up nearly the entire width/height of the video and I will be classifying what hand gesture he or she is doing, I don't need to identify the person and create a bounding box around the person doing the action. I only need to classify video sequences to their class labels.

I will be training on a dataset with individual videos, in which each entire video clip is the particular gesture (So it's a dataset like UCF-101, with video clips corresponding to class labels). But when I am deploying the network, I want the neural network to run on live video. As in how the live video is playing, it should recognize when a gesture has occurred and indicate that it recognized the gesture.

So I was wondering - How can I train the neural network on isolated video sequences in which the entire video clip is the action (like explained above), but run the neural network on live video? For instance, can I use a 3D CNN? Or must I use a 2D CNN with an LSTM network instead, for it to work on live video? My concern is that since a 3D CNN performs the filters across many frames, wouldn't running the CNN on every frame make it very slow? But if I use a 2D CNN with LSTM, will that make it faster? Or will both work fine?

Thank you for your help in advance.

Answer 1

This question actually includes many. I try to answer a couple.

First of all, you need to make sure for your use case to know the non-functional requirements. It is helpful to know that you have a soft real-time case (if harm is done in case of too late predictions it might also be hard real-time)

1. Latency: how much time may pass after the action was made until the prediction is there?
2. Stability of prediction: how fast it's the system allowed to change it's prediction? Once every 0.5s?
3. General hardware, especially memory usage
4. Optimization metric: maybe accuracy does not tell you enough?
5. Evaluation setup: likely, you need to have multiple "evaluation points" per video, e.g. Once after the system had one second, once after it had 1.5,... You will also want to consider the stability of the prediction.

Now to the question about how you can do that. If you didn't try it so far, I recommend looking at optical flow.