There is this claim around that the brain's cognitive capabilities are tightly linked to the way it processes sensorimotor information and that, in this or a similar sense, our intelligence is "embodied". Lets assume, for the sake of argument, that this claim is correct (you may think the claim is too vague to even qualify for being correct, that it's "not even false". If so, I would love to hear your ways of fleshing out the claim in such a way that it's specific enough to be true or false).

Then, since arguably at least chronologically in our evolution, most of our higher-level cognitive capabilities come after our brain's way of processing sensorimotor information, this brings up the question: what is it about the way that our brains function that make them particularly suitable for the processing of sensorimotor information? What makes our brains' architecture particularly suitable for being an information processing unit inside a body?

This is my first question. And what I'm hoping for are answers that go beyond the a fortiori reply "Our brain is so powerful and dynamic, it's great for any task, and so also for processing sensorimotor information".

My second question is basically the same, but, instead of the human brain, I want to ask for neural networks. What are the properties of neural networks that make them particularly suitable for processing the kind of information that is produced by a body?

Here are some of the reasons why people think neural networks are powerful:

  • The universal approximation theorem (of FFNNs)
  • Their ability to learn and self-organise
  • Robustness to local degrading of information
  • Their ability to abstract/coarse-grain/convolute features, etc.

While I see how these are real advantages when it comes to evolution picking its favorite model for an embodied AI, none of them (or their combination) seems to be unique to neural networks. So, they don't provide a satisfactory answer to my question.

What makes a neural network a more suitable structure for embodied AI than, say, having a literal Turing machine sitting inside our head, or any other structure that is capable of universal computation?

For instance, I really don't see how neural networks would be a particularly natural choice for dealing with geometric information. But geometric information is pretty vital when it comes to sensorimotor information, no?


3 Answers 3


To my mind the essential reason why neural networks and the brain are powerful is that they create a hierarchical model of data or of the world. If you ask why that makes them powerful, well, that's just the structure of the world. If you are stalked by a wolf, it's not like its upper jaw will attack you frontally, while his lower jaw will attack you from behind. If you want to respond to the threat with a feasible computational effort, you'll have to treat the wolf as one entity. Providing these kinds of entities or concepts from the raw bits and bytes of input is what a hierarchical representation does.

Now, this is quite intuitive for sensory information: lashes, iris, eyebrow make up an eye, eyes, nose and mouth make up a face and so on. What is less obvious, is the fact that motor control works exactly the same way! Only in reverse. If you want to lift your arm, you'll just lift it. But for your brain to actually realise this move, the high level command has to be broken down into precise signals for every muscle involved. And this is done by propagating the command down the hierarchy.

In the brain these two functions are strongly intertwined. You use constant sensory feedback to adapt your motor control and in many cases you'd be incapable of integrating your stream of sensory data into a coherent representation if you didn't have the additional information of what your body is doing to change that stream of data. Saccades are a good example for that.

Of course this doesn't mean that our cognitive functions are dependent on the processing of sensorimotor information. I would be surprised if a pure thinking machine wouldn't be possible. There is however a specific version of this "embodied intelligence hypothesis" that sounds plausible to me:

Creating high level cognitive concepts with unsupervised learning is a really difficult problem. Creating high level motor representation might be significantly easier. The reason is that there is more immediate useful feedback. I have been thinking about how to provide a scaffolding for the learning of a hierarchy of cognitive concepts and one thing I could imagine is that high level cognitive concepts basically hitch a ride with the motor concepts. Just think of what a pantomime can express with movement alone.

  • $\begingroup$ Dear BlindKungFuMaster, thanks for your reply! I think your point that hierarchical representations of external information are useful in both directions (moving from many bits sensory information to a single concept and from a single motor command to the multitude of motor actions required to implement it) is a very convincing one. But why are neural nets good for hierarchical representation, or rather, what did you have in mind specifically? The "each layer a hierarchy"-idea? Or more along the line of convolutional structural hierarchies? $\endgroup$
    – Paul
    Commented Nov 3, 2016 at 16:02
  • $\begingroup$ I'd also love to hear any progress of yours on providing a "scaffolding for the learning of a hierarchy of cognitive concepts" and "hitching a ride with motor concepts"-idea. I wrote a blog post on something slightly related to this some time ago. Maybe it's interesting to you or you have some criticism. Cheers $\endgroup$
    – Paul
    Commented Nov 3, 2016 at 16:08
  • $\begingroup$ Just the obvious thing, each layer of the net is a level of the hierarchy. Neural networks are inherently hierarchical, because every neuron gets input from many "subordinate" neurons in the lower level. $\endgroup$ Commented Nov 3, 2016 at 17:13

BlindKungFuMaster's answer deals with the hierarchical nature of perception and bodily control, so I'll set that aside and try instead to answer why evolution would use neural networks for animal embodied cognition, and then try to answer if robots of other artificial animals would use the same system.

It's important to focus on animals as a whole, not just humans, because that's how evolution works--like the famous John Gall quote:

A complex system that works is invariably found to have evolved from a simple system that worked.

If you could build a system with five moving parts that does sensorimotor control, but it needs all five parts working in order to function at all, evolution could not build that system except in the rarest of circumstances.

What evolution instead does is slowly extend functional systems. If having one light-sensitive cell connected to one muscle cell makes an organism more likely to survive, then you have the building blocks to add a second layer without inventing any new sorts of cells, because you already have the information-processing connector.

Neural networks are convenient for evolution because their organization matches the hierarchical nature of the problem and the same kind of cell is used everywhere. All you need is dendrites to receive signals, a way to compute the threshold and trigger if the received signal is higher, axons that can make it to other cells, and then branches at the end of the axon to serve as multipliers. You can arbitrarily extend the depth and breadth of the network just by adding more cells.

Neural networks are convenient for artificial sensorimotor control because they give you, in memory, access to lots of intermediate values. They're also convenient for the same reasons evolution found them convenient--we can just say what we expect the structure of the robotic control will look like, provide training data, and then eventually have a robot that works.

But there's lots of robotics where the control system is designed instead of learned. To take a very simple example, one could use machine learning on the thermostat problem, to learn what temperatures require the heater to be turned on and what temperatures require the air conditioner to be turned on. But this would be extra work and a less robust system than just designing the optimal control system ahead of time.

In control theory, there's a concept called adaptive control, where one of the state space parameters for the control system is a property of the system. For example, imagine a satellite; typically we think of the state space of the system as the position and velocity of the satellite in three dimensions, so six total coordinates. There's then a set of differential equations that describe how the satellite will move over time, and what would happen if we used the actuators on the satellite to change its velocity.

But part of those differential equations is the inertia of the satellite. That is, how much fuel we need to expend and how it'll affect the rotation and translation of the satellite depends on where the weight of the satellite is located. And this can change over time, as fuel is consumed or if it wasn't correctly measured to begin with. Adaptive control adds new states to the system to track the inertia, and then simultaneously updates its estimate of the inertia and uses that estimate to plan what controls are necessary to move to a desired position.

You could imagine solving this problem with neural networks, but we can fairly easily calculate the optimal solution from first principles. In that case, we don't need neural network-based control, but the end result will look something like it from the outside.

  • $\begingroup$ Thanks Matthew for your great reply. I have two questions: I take your main point wrt my first question to be that neural networks are "modular" in the sense that you can enlarge a functioning neural network without destroying its functionality, which is also how evolution proceeds. But is this robustness actually the case? I would have thought that as soon as one feeds the output from an additional neuron back into the original network, this may have strong effects on the latter's functionality. I guess that really depends on the activation function, no? $\endgroup$
    – Paul
    Commented Nov 3, 2016 at 15:47
  • $\begingroup$ The second (well, kind of three now) question refers to this paragraph of yours: "Neural networks are convenient for artificial sensorimotor control because...." With the first sentence of that paragraph, do you mean anything other than the learning effect? For the second sentence: But nature never told anyone the structure of any control it would expect, right? Or is this not what yo mean? Thanks! $\endgroup$
    – Paul
    Commented Nov 3, 2016 at 15:51
  • $\begingroup$ If you have a problem where there are simple subproblems, then it does work to train a short network on the first problem, then add a layer and train the network on the second problem, and so on. (For example, starting off with training a two-layer edge detector, then a three-layer shape detector, then...) If you have a hierarchical representation where it's useful to know the lower levels as something besides just a calculation step on the way to knowing the upper levels, then you can expect that sort of system to be successfully built in the wild. $\endgroup$ Commented Nov 3, 2016 at 18:51
  • $\begingroup$ I take your first comment as asking about a different sort of robustness, where I have a functional neural network and, instead of adding more layers that solve different problems, I either add more nodes to the network or add additional layers and attempt to solve the same problem. One expects that the network's performance should only temporarily degrade under this sort of addition, but also that the number of training examples required might grow unnecessarily. $\endgroup$ Commented Nov 3, 2016 at 18:54
  • 1
    $\begingroup$ very helpful clarifications, Matthew, thanks a lot! $\endgroup$
    – Paul
    Commented Nov 4, 2016 at 23:02

what it is about the way that our brains function that make them particularly suitable for the processing of sensorimotor information?

They are an extension of sensory-motor receptors, function could mean any of the hundreds of specific calculations the brain makes, but each one is basically a circuit made out of variations of a basic cell type, with a basic computation, that is a neuron.

What makes our brains' architecture particularly suitable for being an information processing unit inside a body?

I don't think it is helpful to think about inside and outside processing, but rather processing along tracts and nodes,( closer to the receptor, available to consciousness,etc)but leaving aside this distinction, the brain architecture is suitable for processing information ( again what facet of information processing you are referring to is unclear), due to the number of specialized computations that derive from it's evolution.

What are the properties of neural networks that makes them particularly suitable for processing the kind of information that is produced by a body?

A neural network resembles certain parts/circuits of a brain, mainly how information is integrated based on a set of inputs and their frequency, there is variety and nuance in their types, but they all have inputs which in the case of a body are sensory/interneurons cells and outputs; neuron afferents and motor neurons.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .