4
$\begingroup$

Lets say I have a Neural Network with 5 layers, including input and output layer. Each Layer has 5 nodes. Assume the Layers are fully connected, but the 3rd Node in the 2nd Layer is connected to the 5th node in the 4th Layer. All these numbers are chosen at random for the example.

My question is when is the 5th node in the 4th layer fed forward? Lets go through it step by step: the first layer is normally fed forward to the second. the second layer is normally fed forward to the third, but the 3rd node is also fed forward to the 5th node of the 4th layer. So the problem here is, is the 5th node in the 4th layer now fed forward or is it fed forward when the 3rd layer is done being fed forward? The 1st method would mean that the node would get fed forward 2 times and my concern is, if the output is still valid. Further more it would also come to 2 asynchronous outputs and how would these be interpreted?

Because in the Brain, I heard, the neurons are fired when an impulse arrives so this would equal the 1st method.

$\endgroup$
  • $\begingroup$ I'm voting to close this question as off-topic (see scope defined in help center). At present it would rather migrate to StackOverflow to get an answer. It really is a question about programming and runtime execution unfolding. $\endgroup$ – Eric Platon Feb 14 '17 at 3:17
  • $\begingroup$ @EricPlaton but the question is more theoretical and specific for ai. What kind of question would you rather have on this site? $\endgroup$ – Miemels Feb 14 '17 at 6:13
  • $\begingroup$ I understand; I did hesitate to vote for closing. No conclusion yet ! Your question is definitely interesting, but the link to AI is just that it is about NN. NNs are also tools for signal processing, data science, neuro-science models, etc. It means they are not necessarily about intelligence. No definitive answer. NN may just be a way to "make an AI". Because of this possible seasonality of NN, the question becomes border line. The argument here is to ask what better understanding of intelligence and it's making does the question allow? To me, an answer is about wave propagation. $\endgroup$ – Eric Platon Feb 14 '17 at 22:00
  • $\begingroup$ Sorry for the long post. Discussion is open, but let's then move to the chat or Meta. $\endgroup$ – Eric Platon Feb 14 '17 at 22:00
3
$\begingroup$

It is unclear what kind of network your are referring to, there is not a single neural-network model so conceivable both cases could exist and serve some purpose, yet if you are looking for one that emulates nature and real neurons, then you are missing at least 2 ingredients ( time and the mechanisms of resting potentials and refractory periods), which in turn introduce new computations to the neural network.

This is your network graph if I got it right:

enter image description here

The calculation in a neural network without refractory periods and resting potentials without time, would instantaneously modify the weight of your node4 layer 5 (n4-L5) if there is input in column 3:

enter image description here

Additional inputs on other columns would just add up unless you have some other explicit computation on any layer.

If you wanted to emulate a Neuron, each node would need to have a resting potential, that is: a level above which it will fire ( in the above example zero), and a refractory period: a time before it would fire again, as well as a clock to keep it in sync, this would be a crude realtime fascimile:

http://codepen.io/k3no/pen/WRQbYV

enter image description here

A common alternative is to use sequential phases or steps.

Reference/source

Artificial Intelligence: A Modern Approach, by S. Russell and P. Norvig. Deals with a general step approach to A.I.

Gateway to Memory, by Mark A. Gluck and Catherine E. Myers Presents a great and readable introduction to modeling neural networks.

I published the little neural network model in a medium article : Memory and the machine, relevant sources are there.

$\endgroup$
  • $\begingroup$ Thanks. I can't see it clearly on your sketch: is the n4-l3 directly connected to n2-l5? Because it seems that it isn't directly connected. Can you please clarify this to me. $\endgroup$ – Miemels Feb 3 '17 at 20:28
  • $\begingroup$ It is not, if it was connected, there would be an arrow before the node( like there is in n2-L5, unfortunately the connection or edge crosses over n4-L3, so this is a common way of indicating that, along with the +1 on the right side of the node. $\endgroup$ – Keno Feb 3 '17 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.