If I have a set of sensory nodes taking in information and a set of "action nodes" which determine the behavior of my robot, why do I need hidden nodes between them when I can let all sensory nodes affect all action nodes?

(This is in the context of evolving neural network)