3
$\begingroup$

To me, most ANN/RNN related articles don't tell me actually how the network is implemented. I know that in the ANN you'll have multiple neurons, activation function, weights, etc. But, how do you, actually, in each neuron, convert the input to the output?

Putting activation function aside, is the neuron simply doing $\text{input}*a+b=\text{output}$, and try to find the correct $a$ and $b$? If it's true, then how about where you have two neurons and their output ($c$ and $d$) is pointing to one neuron? Do you first multiply $c$ and $d$ then feed it in as input?

$\endgroup$
  • $\begingroup$ Could you give the maths or pseudo-code for "Do you first multiply c and d then feed it in as input?" e.g. Do you mean (using your previous example) c*d*a + b = output? Note that I, and probably many others here, would call that pseudo-code, not maths, although clearly the two things are related. The maths might look like $y = ax + b$ and be concise for large networks if you are happy for all the variables to be vectors or matrices $\endgroup$ – Neil Slater Jul 19 at 11:04
  • $\begingroup$ Yeah I mean (c*d)*a+b=output though the a,b might not be the same as the last sample as they're different neuron $\endgroup$ – Andrew.Wolphoe Jul 19 at 11:12
  • $\begingroup$ @NeilSlater What I'm asking is why the input is consider c*d (if it is) not things like (c+d)? $\endgroup$ – Andrew.Wolphoe Jul 19 at 14:57
  • $\begingroup$ Yes I understand generally what you are asking, I wanted to clarify what you thought the formula might be, because an English description of a maths equation is not precise. Whilst the equation (or code) is precise. You have an answer now, and it looks correct to me. You could write a NN based on multiplying inputs together (and it may even be good at some things), but Simon's answer gives the typical formula for a basic modern NN $\endgroup$ – Neil Slater Jul 19 at 17:21
2
$\begingroup$

Simon Krannig's answer provides the math notation behined exactly what is going on, but since you still seem a bit confused, I've made a visual representation of a neural network using only weights with no activation function. See below: enter image description here

So I'm fairly sure it as you suspected: At each neuron, you take the sum of the inputs of the previous layer multiplied by the weight that connects that specific input to said neuron, where each input has its own unique weight for every one of its outgoing connections.

With a bias, you would do the exact same math as shown in the above image, but once you find the final value (0.2, -0.15, 0.16 and -0.075, the output layer doesn't have a bias) you would add the bias to the total value. So see below for an example including a bias:

enter image description here

NOTE I did not update the outputs at each layer to include the bias because I can't be bothered redrawing this in paint. Just know that the final value for all the nodes with the brown bias haven't carried over to the next layer.

Then, if you were to include an activation function, you would finally take your value and put it through. So including the bias', looking at node 1 of layer 2, it would be (lets pretend your activation function is a sigmoid):

sigmoid((0.4*0.5)+0.2)

and for layer 3 node 2:

sigmoid(((0.6*0.2)+(1.3*-0.15))-0.4)

That is how you would do a forward pass of a simple neural network.

$\endgroup$
4
$\begingroup$

The basic calculation for a single neuron is of the form

$$\sigma\left(\sum_{i} x_i w_i \right),$$

where $x_i$ is the input to the neuron $w_i$ are the neuron-specific weights for every single input and $\sigma$ is the pre-specified activation function. In your terms, and disregarding the activation function, the calculation would turn out to be

$$c\,a_c + d\,a_d + b$$

Note, that the bias term $b$ is just a weight that gets multiplied by the input $1$, thus it appears to have no input.

If you want to develop a further understanding for this, you should try to get familiar with matrix and vector notations and the basic linear algebra that underlies the feed-forward neural networks. If you do, an entire layer of neurons on a whole batch of data will suddenly simply look like this:

$$\sigma(WX)$$

and a FFNN with say 3 layers will look like this:

$$\sigma_{3}(W_3\sigma_2(W_2\sigma_1(W_1X)))$$

$\endgroup$
  • $\begingroup$ I mean in multiple layers ANN, if two neurons (in the same layer) is pointing the same neuron (at the next layer), what will the input of the neuron that's pointed be? Is it the addition of each arrow pointing it? Or the multiply of each arrow pointing it. $\endgroup$ – Andrew.Wolphoe Jul 19 at 14:55
  • $\begingroup$ Also ANN doesn't require addition? (So you use y=as instead of y=ax+b?) Or addition is only required in LSTM? $\endgroup$ – Andrew.Wolphoe Jul 19 at 14:56
  • $\begingroup$ I can see that you are still confused, so I'll try to explain it as well. Let's take for example an input of shape 3 X=[1,2,3] and now if we only have one neuron we will need 3 weights, one for each input right? Lets say W=[4,5,6]. When we multiply the input with the weights we will get 1*4 + 2*5 + 3*6 = 32, and that's the output of the first neuron. Now lets say that we want to add another neuron, instead of 3 weights we will need 6. W=[[4,5,6].[7,8,9]]. The output of the first neuron is the same, but the second one will output 1*7 + 2*8 + 3*9. $\endgroup$ – razvanc92 Jul 19 at 15:23
  • $\begingroup$ @razvanc92 I'm talking at ANN with depth>1. So look at this picture, the bottom section are only multiplying (only one weight for each neuron and top section are following input*a+b=output. And my question are written in the bottom right of each section $\endgroup$ – Andrew.Wolphoe Jul 19 at 22:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.