How exactly is the value of each node determined? Is it the same formula throughout the network?

Question

When it comes to neural networks, it's often only explained what abstract task they do, say for example detect a number in an image. I never understood what's going on under the hood essentially.

There seems to be a common structure of a directed graph, with values in each node. Some nodes are input nodes. Their values can be set. The values of subsequent nodes are then calculated based on those along the edges of the graph until the values for the output nodes are set, which can be interpreted a result.

How exactly is the value of each node determined? I assume that some formula is associated with each node that takes all incoming nodes as input to calculate the value of the node. What formula is used? Is the formula the same throughout the network?

Then I heard that a network has to be trained. I assume that such training would be the process to assign values to coefficients of the formulas used to determine the node values. Is that correct?

In layman's terms, what are the underlying principles that make a neural network work?

Answer 1

I will overly simplify ANNs in order to point how they work. Examples might not be 100% accurate.

In the simplest form, network is trained using the apriori information extracted from the ground truth. This basically means that ANN uses the relation between the input and output.

For instance, if you are to classify shrubs and trees, one of the input could be height and the other could be the width of the tree. Now, if you have only input and output layers, increasing height means increasing chance for the object to be a tree. Thus, input height would have a positive weight connecting to tree output and a negative weight to shrub output. However, as the plant gets wider, the chance of it being a shrub increases. Taller shrubs are wider than shorter ones. Thus input weight would have positive weight connecting to the shrub output. Finally, the chance of being a tree is not affected by the width and thus will have close to 0 weight between this input and output. This network will effectively work like a linear discriminant classifier.

Now instead of assigning weights by hand, you may use a learning algorithm that tries to adjust weights so that the output is correct when the series of input is supplied. Ideally this training algorithm should reach to the conclusion that we have made in the previous example. Most training algorithms are recursive. They supply the inputs multiple times, and in a simple sense, they reward pathways that are correct by increasing their weights and punishes pathways that are causing incorrect answer.

When hidden layers are used in a system, they would be able to correlate input on higher degrees. Thus, as the number of layers get higher, ANN learns the input set much better. However, this does not mean it gets better. If the ANN over fits the input set, it would be affected from the random noise that is in the dataset. This problem is generally referred as memorization. There are learning algorithms that try to minimize memorization and maximize generalization ability. But ultimately, the number of training samples should be high enough so that ANN cannot overfit to the data.

Answer 2

I'll try to do something intuitive; Each node in a neural network is referred to as a neuron. To understand what's going on under the hood of a neural network you only really need to understand an individual neuron.

Now each neuron has a set of inputs (other neurons; they can potentially be the inputs to the network as a whole as well), and each input has a weight associated with it. Every time the network is used, each neuron computes its output as the weighted sum of its inputs passed through some gate (The "Activation Function", a mathematical function designed to get a particular behaviour. For example sigmoid AF takes an input of any size and transforms it into an output in the range [0, 1].) Obviously, this is driven from the inputs to the neural network so that no neuron is computing its outputs before all of the neurons used as its inputs have done the same.

When you refer to the value of a node; there isn't a single value. Each neuron has several weights associated with it as it may be the input to several other neurons, and each of those neurons assigns it a different weight. Instead, it is better to thing of a neural network as a directed graph of nodes (neurons) which are labelled with a particular activation function, and edges (input/output connections) which are labelled with a particular weight. While the structure and activation functions used in the neural networks is a matter of topology design, there are a number of algorithms for designing a ANN for a particular topology.

The most commonly used (and possibly easiest to explain) is backpropagation. In pseudo-Layman's terms we start off with random weights on all edges in the network. We then compute the output of the network for a training set (a set of known input/output pairs). By careful choice of activation function, it is possible to differentiate the error (computed analogously to the expected output minus the actual output of the ANN for each input/output pair) with respect to the weights of the neural network. This allows us to compute a gradient for each weight; the direction in which we can move the weight to reduce the error on the training set. By doing this until we find an optima (a point where all movements increase error), we can find some 'good' configuration of weights for that particular ANN.

There's a nice tutorial on BP here http://www.cse.unsw.edu.au/~cs9417ml/MLP2/BackPropagation.html. The diagram associated with it does nicely to explain my point:

Answer 3

Based on my experience which is that of a beginner.

For a simple neural network such as:

• 2 nodes, indicated by the letters i and j,
• x indicates the output of a node,
• w denotes a weight that connects two nodes.

The output of a given node is of the following form.

Which can be translated as

Applying the activation function (lambda) to the sum of the products of the value of each nodes' of the previous layer and the weights that connect them to the current node.

This activation function can be something like

(This special function is called the sigmoid function.)

If you enter that function in say GeoGebra, you would get the following curve

Clearly this activation function takes any input and outputs a unique number between 0 and 1. Since the function is growing larger and larger the order is preserved.

During the training phase, when the network reaches its termination, we compute the total error of the network which is something that resembles the difference between the output in the training set and the one we obtain from the network.

Obviously, this value decreases each time the output improves.

For each weight of the network, a gradient is computed. This gradient is a number that can be read as the influence of adding a small number to the weight over the total error.

This gradient can be computed from a formula derived from the network structure or it can be computed as simply as trying to add up to the weight and see what happens on the fly.

• if the gradient is positive, it means that adding to the weight will add to the total error, we should subtract,
• if the gradient is negative, it means that adding to the weight will lead to a lesser total error, we should add.

By repeating this a lot of time, the total error will reach its minimum.

Finally a thing I didn't know, don't forget to switch inputs between iterations of this process. If you don't, your network will only be properly trained against the last item it processed.

I hope this helped a little. Please write your suggestions in the comment. As you probably guessed I'm not native English speaker.

I recommend reading Neural Networks, A Visual Introduction For Beginners by Michael Taylor.