# Using Convolutional Neural Networks for movement classification

I have programmed my first network for the MNIST dataset. I was wondering what the first approach would be to recognize certain movements.

I have read about that the time dimension should be considered for solving such problems, but that's where I am stuck.

You have to here a combination of two networks,

1. Convolutional neural networks for processing the image.
2. Recurrent neural networks for handling the time feature.

One can indeed use CNNs to classify movements by doing exactly what the question stated, by using a tensor in $$\mathbb{R}^4$$ to cover the dimensions $$h, v, d, t$$ for horizontal index, vertical index, pixel depth layer index, and time (or frame number) instead of the $$\mathbb{R}^3$$ to cover only still frames with $$h, v, d$$. However, this is not the best practice, for efficiency reasons. The number of frames required to capture patterns with longer duration events is prohibitive.

The Long-Short Term Memory (LSTM) approach, an improvement over RNNs, is one of the most common solutions in current use. It is specifically designed to handle time series efficiently. Variants also exist, as well as other strategies derived from RNNs such as Gated Recurrent Unit (GRU) designs and Residual Attention Networks designs. This is an area of machine learning where a continuous stream of minor discoveries leading to incremental improvements makes anything one learns stale in a few years.

These approaches do not entirely discard CNN technology. Most either integrate CNN technology, as with ConvGRU, or are used in conjunction with CNN and max pooling layers.

To gauge what is both up and coming and what is practical for consumer and industrial use today, further reading of scholarly articles is recommended, using the above terms as search phrases, followed by downloading some of the open source implementations of the algorithms and approaches. The hands-on experience will help solidify concepts and show the current state of usability for each theoretical idea.

A good answer depends on your input, your resources and how much preprocessing you're willing to do.

Also, what exactly do you mean with "recognize certain movements"?

Probably the most common thing to have is:

• A desktop computer with a high end GPU for inference
• Gesture recognition

Then you can try to detect the hands and the face with a CNN in each frame. Find a representation is the position is those. Then you can use a RNN for clarifying the sequence is hand/head positions.

Have a look at optical flow

Datasets: UCF Sport; Hollywood 2; Sports-1M

You can treat the problem as a 3D convolution problem, just build a 3D convolutional network where the third dimension is the timeframe of the video.