The Degree to Which Inhibition is in Common Use
What could loosely be considered inhibitory effect occurs in MLPs (multilayer perceptrons) as they are normally designed and implemented already.
The gradient descent scheme implemented within a larger back propagation algorithm can produce a parameter adjustment delta that is either positive or negative.
- A positive value decreases the attenuation of that parameter's signal path, thereby increasing the signal strength there.
- A negative value increases the attenuation of that path, thereby
decreasing signal strength through that connection.
A decrease in a parameter's value as a result of back propagation bears some similarity to the inhibition of a neural signal path, however, you may already be aware of the significant differences in the signaling between biological neurons and the signalling between layers in the type of artificial networks commonly in machine learning.
The term inhibition is, as mentioned, only loosely applicable.
- One cannot inhibit a pulse through a MLP because there ARE NO PULSES in MLPs.
- One cannot alter the signal attenuation between neurons by varying a numeric parameter either, since there is no numeric parameter array in a biological net.
Stimulation and inhibition in the brains of mammals are also different in that neuro-chemistry impacts the network regionally, so the terms stimulation and inhibition are a bit ambiguous, since we have agonists and antagonists ranging from dopamine to serotonin and from cannabanoids to oxytocin receptors and from endorphins to other classes.
Changes from Former Textbook Themes
The former thinking was that a pulse travelling through a biological signal path strengthened that connection. No one in neurology research adheres to that simplistic a conception today.
For example, it is know that a signal pathway may be in common use but may close down by repeated sharp pains following its use. Although I am not well trained in electro-chemical processes in neural pathways, I recall in vitro experiments supporting that this is neither neuro-plastic nor electrical, that it is related to regional chemical feedback.
The current view of addiction as a brain disease is that a breakdown of the interrelationship between chemical state change and learned inhibition or transmission is causal. Inhibition or transmission is no longer decided upon based on organism survival and socialization but on the addictive stimuli, leading to behavioral dysfunction.
It may be useful to point out that, stimulation and inhibition are not strictly antonyms. The opposite of inhibiting a signal is the transmission of it. The opposite of stimulation is the lack of stimulation (no signal).
Attempting Analogy in Largely Dissimilar Circuit Models
It may not be an aid to general understanding to draw parallels between ReLU activation functions and the functions of synapses with their sensitivity to regional brain chemistry and with cell-level retention functions orchestrated by organelles.
Neural nets are not neural. They are a mathematical conception sharing only the ideal of learning as convergence on some ideal network behavior. Nothing else of significance is in common.
Adding to the Disparity Between MLPs and Biology
In a sense inhibition in the brain occurs at multiple architectural levels, inside the cell, between cells, and over structures of cells, and the alignment of pulses temporally (in the time domain) is not simulated at all in conventional machine learning constructs.
Some researchers have deviated entirely from the multilayer perceptron design and favored a pulse based system that requires specialized hardware. Follow the money there. It is not an inexpensive research avenue yet. But it may become one if they have success.
Curvature in Functions
Brief note on terminology: Nth degree polynomials fall under linear algebra, so the best term to use is 'curved functions' so as to not fall into the ambiguity of the term non-linear.
Nonetheless, you are correct that there are non-linearities of different types in biological neural circuits. Potential change is not only curved, but its function's curvature changes quickly. It is temporally sensitive.
On the longer time frame, the memory in a cell forms through neural plasticity and the cell behavior changes internally (within the cell membrane) employing cytoplasm and the suspended organelles. That memory function also attenuates at a roughly inverse exponential rate with respect to time but some have hypothesized based in empirical evidence that forgotten cellular function can be recalled. Again, this is at the cellular level.
The second non-linearity is not a sum of potentials. The surface of the function that aggregates incoming signals is not flat. It is curved. Also, as mentioned, the temporal alignment presents a complexity, since perfect pulse alignment is not treated the same as pulses not perfectly aligned in time.
(I brought up the absurdity of using an additive adjustment in MLP back propagation in a question I wrote for this site. The responses to the challenge to the status quo not particularly well understood by the majority of machine learning practitioners were not outstanding.)
Linear Thinking Prevails Currently
To a large degree linear thinking (in the wider sense of the term) pervades mainstream machine learning and data science today, the activation functions being a notable and welcomed exception.
Over time, I expect that will improve. I see current leading edge research going beyond that linear thinking and considering short and long term memory as in the LSTM and attention based networks, the simulation of the curved surfaces that represent pulse propagation in mammalian nets and the consideration of various applications of exponential decay here and there in the latest literature.
Gratitude for the Question
Questions like this one may help widen the mainstream understanding too.
