# What is the purpose of hidden nodes in neural network?

If I have a set of sensory nodes taking in information and a set of "action nodes" which determine the behavior of my robot, why do I need hidden nodes between them when I can let all sensory nodes affect all action nodes?

(This is in the context of evolving neural network)

• Look for the idea called "Hierarchical representation" Oct 24 '16 at 6:55

A feed forward neural network without hidden nodes can only find linear decision boundaries. However, most of the time you need non-linear decision boundaries. Hence you need hidden nodes with a non-linear activation function. The more hidden nodes you have, the more data you need to find good parameters, but the more complex decision boundaries you can find.

• It appears that the topic is much more complex than I initially thought. What field of study is this in, and is there a name for this specific aspect of neural networking? Nov 1 '16 at 3:44
• @user289661 This is one of the most basic aspects of artificial neural networks. It's in machine learning. Nov 1 '16 at 10:41
• And you generally want more than one hidden layer, because while a sufficiently large network can capture any function to an arbitrary precision, for certain computations a shallow network would need exponentially more nodes than a one layer deeper network that does the same. Nov 1 '16 at 21:12
• @Peteris While I agree that you generally want more than one hidden layer, I wonder if you have a source for the claim that a shallow network needs exponentially more nodes than a one layer deeper network. Nov 1 '16 at 22:05
• @MartinThoma arxiv.org/pdf/1603.00988.pdf or arxiv.org/pdf/1509.08101.pdf are some sources; the particular proofs of course are different for different types of functions; It generally starts with Hastad 1987 "Computational limitations of small-depth circuits" and in general this concept is the counterpoint to the universal approximator theorem - while yes, even the simplest architectures can implement anything, for certain functions they require an unreasonable amount of resources. Nov 1 '16 at 23:17

Neural Networks are very good approaches for robots. The main function of Neural Net is to model the interdependence between all the features. Now this can be done manually by selecting possible combinations of features between themselves upto a certain degree. But this approach has drawbacks:

• It is tedious to go about selecting features.
• It costs time and additional computer resources to calculate the values of the new features you have introduced.
• Since you cannot visualize data more than 3-D you cannot be absolutely sure that your selected features are enough to model your problem.

Now if you use an NN, the NN will automatically select the combination of features (provided it has enough hidden nodes) by adjusting the weights of connections between and the features and nodes. The main advantages of this approach are:

• You don't have to manually select the feature combinations.
• If data is still not fitting you can easily increase or decrease the number of nodes without needing to modify the whole network.
• Also it will be computationally efficient since you don't have to calculate values of factors that don't matter to the problem.

Hope this is what you were looking for!

Normally one node/layer applies linear fitting of the the input to the hypothesis, in other words uses linear function ($$y = ax + b$$). Adding layers chains liner functions, potentially allowing fitting higher order functions. A great explanation can be found here.

The term hidden nodes refers to the cells of inner layers of artificial networks are not exposed for connectivity outside of their connectivity within the network. Their values can be read and visualized, but the network function is not dependent upon such tapping of signals internal to the network. Neither are the input of inner layers and their hidden cells connected to network inputs nor are the outputs of inner layers and their hidden cells connected to network outputs.

The purpose of inner layers is related to network flexibility and thus accuracy of the functional approximation that may be achieved. An activation function by itself rarely approximates the desired mapping between input values and output values that is optimally achieved by training.

Attenuated substitution of one parallel activation function into another leads to greater functional flexibility. Attenuation is achieved by multiplying the vector of layer activation function outputs by a matrix of parameters and feeding the resulting product vector into the next layer's activation function units. In combination with a convergence strategy like gradient descent and a corrective distribution strategy like back-propagation to update the parameters, the flexibility in functionality can be directed to achieve higher resulting accuracy by the end of training.

There are no action layers or action cells. The cells in an artificial network do not perform actions other than the evaluation of their activation function (and gating in the case of gated units). The output of the network may lead to actions if the network is a component in a controller, in which case the output of the network is connected to other system components so that action is controlled.