2
$\begingroup$

Something I like about neural network AI is that we already have a blueprint for it - we know in great detail how different types of neurons work, and we don't have to invent the AI, we can just replicate what we already know works (neural networks) in a simplified way and train it.

So what's confusing me right now is why many popular neural models I've seen when studying neural networks include both inhibitory and stimulating connections. In real neural networks, from my understanding, there is no negative signal being transferred, rather the signals sent between neurons are comparable to values between 0.1 and 1. There's no mechanic for an inverse (inhibiting) signal to be sent.

For example, in this network (seen overlaying a snake being simulated), the red lines represent inhibiting connections, and the blue lines represent stimulating neurons:

enter image description here

Is this just an inconsequential detail of neural network design, where there's really no significant difference between a range of 0 to 1 and a range of -1 to 1? Or is there a reason that connections in our simulated neural networks benefit from being able to a express a range from -1 to 1 rather than just 0 to 1?

Improve this question
$\endgroup$
1
  • $\begingroup$ If by real life neural networks, you mean the squishy kind in somebodies skull, those do have inhibitory connection. If not, what is the difference between "real life neural networks" and "virtual neural networks"? $\endgroup$
    – BlindKungFuMaster
    May 18, 2017 at 6:41

2 Answers 2

Reset to default
2
$\begingroup$

I think you are confused. The reason why some neural neural networks have neurons with an output in range (-1, 1) all depends on the activation function used. Some networks even have neurons with an output range of (-Infinity, +Infinity) ( aka the identity function ).

I advise you to take a look at this list: activation functions w/ ranges.

Nowadays a lot of neural networks have a mixture of different activation functions. E.g. combining ReLU with Sigmoid or TanH.

This neural network library even shows that for most simple problems, neural networks prefer a variety of activation functions.

The red lines just mean that that connection has a low value in comparison with other connections.

There is no rule of thumb for activation functions, some problems benefit from a certain function, some don't.

Improve this answer
$\endgroup$
2
$\begingroup$

From what I've read on deep nets and neuroscience, your interpretation is backwards. Most 'neurons' in deep nets use RELU activations, which is strictly excitatory, while mammal brains are composed of both excitatory and inhibitory synapses/neurons.

https://en.wikipedia.org/wiki/Excitatory_postsynaptic_potential

https://en.wikipedia.org/wiki/Inhibitory_postsynaptic_potential

I think the reason for strictly excitatory activations like RELU in deep nets is that they provide a continuous gradient for backpropagation during learning. Inhibitory activations make the gradient discontinuous, greatly complicating the propagation of feedback/error.

Improve this answer
$\endgroup$
1
  • $\begingroup$ But sigmoids have negativ output as well, which acts inhibitory. For some problems you need activation functions that can be negativ. I think ReLU are widely used in vision because vision works largely by combining features, not so much by determining features that definitely shouldn't be there for certain classes. $\endgroup$
    – BlindKungFuMaster
    May 19, 2017 at 10:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .