Let's say I have a neural network with 5 layers, including the input and output layer. Each layer has 5 nodes. Assume the layers are fully connected, but the 3rd node in the 2nd layer is connected to the 5th node in the 4th layer. All these numbers are chosen at random for the example.

When is the 5th node in the 4th layer fed forward?

Let's go through it step by step. The first layer is normally fed forward to the second. The second layer is normally fed forward to the third, but the 3rd node is also fed forward to the 5th node of the 4th layer.

At this point, is the 5th node in the 4th layer now fed forward, or is it fed forward when the 3rd layer is done being fed forward? The 1st method would mean that the node would get fed forward 2 times and my concern is, if the output is still valid.

Furthermore, it would also come to 2 asynchronous outputs and how would these be interpreted?

Because in the brain, I heard, the neurons are fired when an impulse arrives so this would equal the 1st method.


It is unclear what kind of network your are referring to, there is not a single neural-network model so conceivable both cases could exist and serve some purpose, yet if you are looking for one that emulates nature and real neurons, then you are missing at least 2 ingredients ( time and the mechanisms of resting potentials and refractory periods), which in turn introduce new computations to the neural network.

This is your network graph if I got it right:

enter image description here

The calculation in a neural network without refractory periods and resting potentials without time, would instantaneously modify the weight of your node4 layer 5 (n4-L5) if there is input in column 3:

enter image description here

Additional inputs on other columns would just add up unless you have some other explicit computation on any layer.

If you wanted to emulate a Neuron, each node would need to have a resting potential, that is: a level above which it will fire ( in the above example zero), and a refractory period: a time before it would fire again, as well as a clock to keep it in sync, this would be a crude realtime fascimile:


enter image description here

A common alternative is to use sequential phases or steps.


Artificial Intelligence: A Modern Approach, by S. Russell and P. Norvig. Deals with a general step approach to A.I.

Gateway to Memory, by Mark A. Gluck and Catherine E. Myers Presents a great and readable introduction to modeling neural networks.

I published the little neural network model in a medium article : Memory and the machine, relevant sources are there.

  • $\begingroup$ Thanks. I can't see it clearly on your sketch: is the n4-l3 directly connected to n2-l5? Because it seems that it isn't directly connected. Can you please clarify this to me. $\endgroup$
    – Miemels
    Feb 3 '17 at 20:28
  • $\begingroup$ It is not, if it was connected, there would be an arrow before the node( like there is in n2-L5, unfortunately the connection or edge crosses over n4-L3, so this is a common way of indicating that, along with the +1 on the right side of the node. $\endgroup$
    – Keno
    Feb 3 '17 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.