How to achieve recognition of postures and gestures?

Question

I'd like to investigate the possibility of achieving similar recognition as it's in Honda's ASIMO robot^p.22 which can interpret the positioning and movement of a hand, including postures and gestures based on visual information.

Here is the example of an application of such a recognition system.

^{Image source: ASIMO Featuring Intelligence Technology - Technical Information (PDF)}

So, basically, the recognition should detect an indicated location (posture recognition) or respond to a wave (gesture recognition), like a Google car does it (by determining certain patterns).

Is it known how ASIMO does it, or what would be the closest alternative for postures and gestures recognition to achieve the same results?

Answer 1

Just to add some discourse; this is actually an incredibly complex task, as gestures (aka kinematics) function as an auxiliary language that can completely change the meaning of a sentence or even a single word. I recently did a dissertation on the converse (generating the correct gesture from a specific social context & linguistic cues). The factors that go into the production of a particular gesture include the relationship between the two communicators (especially romantic connotations), the social scenario, the physical context, the linguistic context (the ongoing conversation, if any), a whole lot of personal factors (our gesture use is essentially a hybrid of important individuals around us e.g. friends & family, and this is layered under the individual's psychological state). Then the whole thing is flipped around again when you look at how gestures are used completely differently in different cultures (look up gestures that swear words in other cultures for example!). There are a number of models for gesture production but none of them captures the complexity of the topic.

Now, that may seem like a whole lot of fluff that is not wholly relevant to your question, but my point is that ASIMO isn't actually very 'clever at this. AFAIK (I have heard from a visualization guy that this is how he thinks they do it) they use conventional (but optimized) image recognition techniques trained on a corpus of data to achieve recognition of particular movements. One would assume that the dataset consists of a series of videos/images of gestures labelled with that particular gesture (as interpreted by a human), which can then be treated as a machine learning problem. The issue with this is that it does not capture ANY of the issues I mentioned above. Now if we return to the current best interpretation of gesture that we have (that it is essentially auxiliary language in its own right), ASIMO isn't recognizing any element of language beyond the immediately recognizable type, 'Emblems'.

'Emblems' are gestures that have a direct verbal translation, for example in English-based cultures, forming a circle with your thumb and index finger translates directly to 'OK'. ASIMO is therefore missing out on a huge part of the non-verbal dictionary (illustrators, affect displays, regulators and adapters are not considered!), and even the part that it is accessing is based on particular individuals' interpretations of said emblems (e.g. someone has sat down and said that this particular movement is this gesture which means this), which as we discussed before is highly personal and contextual. I do not mean this in criticism of Honda; truth be told, gesture recognition and production are in my opinion one of the most interesting problems in AI (even if it's not the most useful) as it is a compound of incredibly complex NLP, visualization and social modelling problems!

Hopefully, I've provided some information on how ASIMO works in this context, but also on why ASIMO's current process is flawed when we look at the wider picture.

Answer 2

There is some research on this topic. See, for example, the papers Robot Identification and Localization with Pointing Gestures (2018) and Proximity Human-Robot Interaction Using Pointing Gestures and a Wrist-mounted IMU (2019), by Boris Gromov et al., where the human is assumed to possess an inertial measurement unit (IMU) attached to the arm

Answer 3

It's not a difficult task, first of all you have to locate the body parts such as arms,head... you can do it using different approaches for example using cascadeclassifier or a well trained CNN.
After that you can use different techniques, one could be an ANN trained on the keypoints of the different body parts (this is the easiest approach) or a CNN (good approach but you need a lot of training). To indicate the location after you have determined the position of the head (and the eyes to) and hands, you can simply calculate the orientation of those parts, and then you can get a general position where those orientation are pointing to.