I am going to train a deep learning model to classify hand gestures in video. Since the person will be taking up nearly the entire width/height of the video and I will be classifying what hand gesture he or she is doing, I don't need to identify the person and create a bounding box around the person doing the action. I only need to classify video sequences to their class labels.
I will be training on a dataset with individual videos, in which each entire video clip is the particular gesture (So it's a dataset like UCF-101, with video clips corresponding to class labels). But when I am deploying the network, I want the neural network to run on live video. As in how the live video is playing, it should recognize when a gesture has occurred and indicate that it recognized the gesture.
So I was wondering - How can I train the neural network on isolated video sequences in which the entire video clip is the action (like explained above), but run the neural network on live video? For instance, can I use a 3D CNN? Or must I use a 2D CNN with an LSTM network instead, for it to work on live video? My concern is that since a 3D CNN performs the filters across many frames, wouldn't running the CNN on every frame make it very slow? But if I use a 2D CNN with LSTM, will that make it faster? Or will both work fine?
Thank you for your help in advance.