Based on my experience which is that of a beginner.
For a simple neural network such as:
- 2 nodes, indicated by the letters
x indicates the output of a node,
w denotes a weight that connects two nodes.
The output of a given node is of the following form.
Which can be translated as
Applying the activation function (lambda) to the sum of the
products of the value of each nodes' of the previous layer and the weights
that connect them to the current node.
This activation function can be something like
(This special function is called the sigmoid function.)
If you enter that function in say GeoGebra, you would get the following
Clearly this activation function takes any input and outputs
a unique number between 0 and 1. Since the function is
growing larger and larger the order is preserved.
During the training phase, when the network reaches its termination,
we compute the total error of the network
which is something that resembles the difference between
the output in the training set and the one we obtain from
Obviously, this value decreases each time the output improves.
For each weight of the network, a gradient is computed.
This gradient is a number that can be read as the influence
of adding a small number to the weight over the total error.
This gradient can be computed from a formula derived from
the network structure or it can be computed as simply as
trying to add up to the weight and see what happens on the fly.
- if the gradient is positive, it means that adding to the weight
will add to the total error, we should subtract,
- if the gradient is negative, it means that adding to the weight
will lead to a lesser total error, we should add.
By repeating this a lot of time, the total error will reach its
Finally a thing I didn't know, don't forget to switch inputs between
iterations of this process. If you don't, your network will only be
properly trained against the last item it processed.
I hope this helped a little. Please write your suggestions in the comment.
As you probably guessed I'm not native English speaker.
I recommend reading Neural Networks, A Visual Introduction For Beginners
by Michael Taylor.