A paper from machinelearningmastery.com on human activity recognition states that 1D convolutional neural networks work the best on classification of human activities using data from accelometer. But, according to me, human activities like swinging the arm are sequential actions and they require LSTMs. So, they one should be more efficient, CNNs or LSTMs. Or, in other words, is spatial learning required or sequence learning?

  • $\begingroup$ Neural networks alone are not enough for human activity recognition. What is used in real systems is “grammar based human activity recognition”. A famous EU FP7 program was Poeticon+. The idea is, to record motion capture data and search in the activity language for matchings. To find similar activities, neural networks are used which can be LSTM. Unfortunately, in some weak papers this inbetween step with a grammar is not mentioned. $\endgroup$ – Manuel Rodriguez Nov 24 '18 at 19:57
  • $\begingroup$ So, is it fine if we use the hybrid of CNN and LSTM that is Convolutional LSTM networks ? $\endgroup$ – Shubham Panchal Nov 25 '18 at 1:46
  • $\begingroup$ In a hybrid neural network the CNN can be used for pixel based image-recognition while LSTM becomes the task of symbolic recognition with a grammar. $\endgroup$ – Manuel Rodriguez Nov 25 '18 at 10:24
  • $\begingroup$ @ManuelRodriguez Beyond Short Snippets: Deep Networks for Video Classification does not use a grammar. $\endgroup$ – Martin Thoma Nov 25 '18 at 11:43

Human activity recognition, for example as done with the Sports-1M Dataset, is a classification task. Given a video, say which activity / action was executed.

So it is a special case about video classification.

human activities like swinging the arm are sequential actions and they require LSTMs

This is wrong. As a simple example, think of handwriting recognition as done on write-math.com: I receive a sequence of points (x, y) -- the pen-tip -- and classify which symbol is written. So it is a sequence, but still there a plenty of ways to use a simple multi-layer perceptron (MLP) for it. Two big groups of approaches:

  • Feature-Engineering: Instead of giving an arbitrary number of coordinates, one can normalize it to a fixed amount. This is what is currently done on write-math.com and works pretty well (see my Bachelors thesis)
  • Space Transformation: Similar to the feature engineering, but more extreme. Instead of looking at (variants of) coordinates, one can look at the image which is there in the end. This way one can apply CNNs.

You can do similar things for movies:

  • Multiple Classifications: Classify based on single frames and average the results
  • Feature-Engineering: Normalize the length to a fixed number; interpret the fixed-length video as an image with more channels.
  • Use CNNs as feature extractors / for dimensionality reduction and LSTM for the sequence classification.

You might be interested in Beyond Short Snippets: Deep Networks for Video Classification

Efficiency vs Effectiveness

Model A is more effective than model B, if its optimizarion criterion is better.

Model A is more efficient than model B, if it uses less resources (e.g. memory) to achieve the same optimization value (e.g. same accuracy / MSE)

| improve this answer | |
  • $\begingroup$ Feature extraction for human activity recognition is not a simple classification task. The features are ordered in a gantt chart and the events have a timecode. Compared to an optical character recognition, it is a different kind of task to recognize the pattern. The linked paper “Beyond Short Snippets Deep Networks for Video Classification” can't be recommended because it's not about a concrete problem, but gives only abstract information about recognition in a dataset. $\endgroup$ – Manuel Rodriguez Nov 25 '18 at 12:01
  • $\begingroup$ Feature extraction is not a classification task at all. I never claimed that. OCR is also different from what I wrote in the answer. The paper I linked is evaluated on Sports-1M and UCF-101. What is not concrete about that? $\endgroup$ – Martin Thoma Nov 25 '18 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.