In the brain, some synapses are stimulating and some inhibiting. In the case of artificial neural networks, ReLU erases that property, since in the brain inhibition doesn't correspond to a 0 output, but, more precisely, to a negative input.

In the brain, the positive and negative potential is summed up, and, if it passed the threshold, the neuron fires.

There are 2 main non-linearities which came to my mind in the biological unit:

  • potential change is more exponential than linear: small amount of ion channels is sufficient to start a chain-reaction of other channels activation's - which rapidly change global neuron's potential.

  • the threshold of the neuron is also non-linear: neuron fires only when the sum of its positive and negative potentials passed given (positive) threshold

So, is there any idea how to implement negative input to the artificial neural network?

I gave examples of non-linearities in biological neurons because the most obvious positive/negative unit is just a linear unit. But, since it doesn't implement non-linearity, we may consider implementing non-linearities somewhere else in the artificial neuron.


3 Answers 3


Principles of Computational Modelling in Neuroscience by David Sterratt, Bruce Graham, Andrew Gillies and David Willshaw discuss it in Chapter 7 (The synapse) and also in Chapter 8 (Simplified models of neurons). Especially in chapter 8, they discuss how to add excitatory or inhibitory synapses to integrate and fire neuron.

There are various ways to add inhibitory synapse: either substrate voltage, inject negative current.


In biology, when the presynaptic releases a neurotransmitter (a positive amount of them, obviously), this neurotransmitter reaches the postsynaptic vesicles causing an excitatory (depolarization) or inhibitory (hyperpolarization) effect, depending on the kind of postsynaptic vesicle in next cell dendrites. If the total amount of depolarization (all dendrites) is enough bigger than hyperpolarization, the neuron triggers an action potential or similar signal, continuing with the chain.

In the artificial NeuralNet parallelism, when the activation function of previous layer provides an output (say positive one) this value is multiplied by the weights of next layer cell. If the weight is positive, the effect is excitatory, if the weight is negative, the effect is inhibitory.

Thus, these two models are functionally equivalent (same excitatory/inhibitory target is covered), just make the analogy between kind of postsynaptic vesicle with the input weight sign of the artificial neuron.


The Degree to Which Inhibition is in Common Use

What could loosely be considered inhibitory effect occurs in MLPs (multilayer perceptrons) as they are normally designed and implemented already.

The gradient descent scheme implemented within a larger back propagation algorithm can produce a parameter adjustment delta that is either positive or negative.

  • A positive value decreases the attenuation of that parameter's signal path, thereby increasing the signal strength there.
  • A negative value increases the attenuation of that path, thereby decreasing signal strength through that connection.

A decrease in a parameter's value as a result of back propagation bears some similarity to the inhibition of a neural signal path, however, you may already be aware of the significant differences in the signaling between biological neurons and the signalling between layers in the type of artificial networks commonly in machine learning.

The term inhibition is, as mentioned, only loosely applicable.

  • One cannot inhibit a pulse through a MLP because there ARE NO PULSES in MLPs.
  • One cannot alter the signal attenuation between neurons by varying a numeric parameter either, since there is no numeric parameter array in a biological net.

Stimulation and inhibition in the brains of mammals are also different in that neuro-chemistry impacts the network regionally, so the terms stimulation and inhibition are a bit ambiguous, since we have agonists and antagonists ranging from dopamine to serotonin and from cannabanoids to oxytocin receptors and from endorphins to other classes.

Changes from Former Textbook Themes

The former thinking was that a pulse travelling through a biological signal path strengthened that connection. No one in neurology research adheres to that simplistic a conception today.

For example, it is know that a signal pathway may be in common use but may close down by repeated sharp pains following its use. Although I am not well trained in electro-chemical processes in neural pathways, I recall in vitro experiments supporting that this is neither neuro-plastic nor electrical, that it is related to regional chemical feedback.

The current view of addiction as a brain disease is that a breakdown of the interrelationship between chemical state change and learned inhibition or transmission is causal. Inhibition or transmission is no longer decided upon based on organism survival and socialization but on the addictive stimuli, leading to behavioral dysfunction.

It may be useful to point out that, stimulation and inhibition are not strictly antonyms. The opposite of inhibiting a signal is the transmission of it. The opposite of stimulation is the lack of stimulation (no signal).

Attempting Analogy in Largely Dissimilar Circuit Models

It may not be an aid to general understanding to draw parallels between ReLU activation functions and the functions of synapses with their sensitivity to regional brain chemistry and with cell-level retention functions orchestrated by organelles.

Neural nets are not neural. They are a mathematical conception sharing only the ideal of learning as convergence on some ideal network behavior. Nothing else of significance is in common.

Adding to the Disparity Between MLPs and Biology

In a sense inhibition in the brain occurs at multiple architectural levels, inside the cell, between cells, and over structures of cells, and the alignment of pulses temporally (in the time domain) is not simulated at all in conventional machine learning constructs.

Some researchers have deviated entirely from the multilayer perceptron design and favored a pulse based system that requires specialized hardware. Follow the money there. It is not an inexpensive research avenue yet. But it may become one if they have success.

Curvature in Functions

Brief note on terminology: Nth degree polynomials fall under linear algebra, so the best term to use is 'curved functions' so as to not fall into the ambiguity of the term non-linear.

Nonetheless, you are correct that there are non-linearities of different types in biological neural circuits. Potential change is not only curved, but its function's curvature changes quickly. It is temporally sensitive.

On the longer time frame, the memory in a cell forms through neural plasticity and the cell behavior changes internally (within the cell membrane) employing cytoplasm and the suspended organelles. That memory function also attenuates at a roughly inverse exponential rate with respect to time but some have hypothesized based in empirical evidence that forgotten cellular function can be recalled. Again, this is at the cellular level.

The second non-linearity is not a sum of potentials. The surface of the function that aggregates incoming signals is not flat. It is curved. Also, as mentioned, the temporal alignment presents a complexity, since perfect pulse alignment is not treated the same as pulses not perfectly aligned in time.

(I brought up the absurdity of using an additive adjustment in MLP back propagation in a question I wrote for this site. The responses to the challenge to the status quo not particularly well understood by the majority of machine learning practitioners were not outstanding.)

Linear Thinking Prevails Currently

To a large degree linear thinking (in the wider sense of the term) pervades mainstream machine learning and data science today, the activation functions being a notable and welcomed exception.

Over time, I expect that will improve. I see current leading edge research going beyond that linear thinking and considering short and long term memory as in the LSTM and attention based networks, the simulation of the curved surfaces that represent pulse propagation in mammalian nets and the consideration of various applications of exponential decay here and there in the latest literature.

Gratitude for the Question

Questions like this one may help widen the mainstream understanding too.

  • $\begingroup$ Thanks for good response/discussion. I wonder what is possible to achieve today, without special-designed chip. I feel that spike-time-dependency is the key prerequisite before implementing any inhibitory principle and this in turn refers more to hardware than to the software. So we're stuck here for years or even decades. I do not believe much in LSTM approach because, those architectures are too naive. $\endgroup$
    – Ziemo
    Commented Aug 13, 2018 at 9:46
  • $\begingroup$ Maybe my question should be viewed from the perspective of axioms which ANN should follow if we want to implement something much more biological in the future rather than request for exhaustive answer. From this angle I think that spike-time-dependency (which I believe you called - quite intuitively - the pulse propagation) should be the very first axiom. $\endgroup$
    – Ziemo
    Commented Aug 13, 2018 at 9:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .