Does the human brain use a specific activation function?
I've tried doing some research, and as it's a threshold for whether the signal is sent through a neuron or not, it sounds a lot like ReLU. However, I can't find a single article confirming this. Or is it more like a step function (it sends 1 if it's above the threshold, instead of the input value)?
The thing you were reading about is known as the action potential. It is a mechanism that governs how information flows within a neuron.
It works like this: Neurons have an electrical potential, which is a voltage difference inside and outside the cell. They also have a default resting potential, and an activation potential. The neuron tends to move towards the resting potential if it is left alone, but incoming electric activations from dendrites can shift its electric potential.
If the neuron reaches a certain threshold in electric potential (the activation potential), the entire neuron and its connecting axons goes through a chain reaction of ionic exchange inside/outside the cell that results in a "wave of propagation" through the axon.
TL;DR: Once a neuron reaches a certain activation potential, it electrically discharges. But if the electric potential of the neuron doesn't reach that value then the neuron does not activate.
Does the human brain use a specific activation function?
IIRC neurons in different parts of the brain behave a bit differently, and the way this question is phrased sounds as if you are asking if there is a specific implementation of neuronal activation (as opposed to us modelling it).
But in general behave relatively similar to each other (Neurons communicate with each other via neurochemicals, information propagates inside a neuron via a mechanism known as the action potential...) But the details and the differences they cause could be significant.
There are various biological neuron models, but the Hodgkin-Huxley Model is the most notable.
Also note that a general description of neurons don't give you a general description of neuronal dynamics a la cognition (understanding a tree doesn't give you complete understanding of a forest)
But, the method of which information propagates inside a neuron is in general quite well understood as sodium / potassium ionic exchange.
It (activation potential) sounds a lot like ReLU...
It's only like ReLU in the sense that they require a threshold before anything happens. But ReLU can have variable output while neurons are all-or-nothing.
Also ReLU (and other activation functions in general) are differentiable with respect to input space. This is very important for backprop.
This is a ReLU function, with the X-axis being input value and Y-axis being output value.
And this is the action potential with the X-axis being time, and Y being output value.
The brains of mammals do not use an activation function. Only machine learning designs based on the perceptron multiply the vector of outputs from a prior layer by a parameter matrix and pass the result statelessly into a mathematical function.
Although the spike aggregation behavior has been partly modeled, and in far more detail than the 1952 Hodgkin and Huxley model, all the models require statefulness to functionally approximate biological neurons. RNNs and their derivatives are an attempt to correct that shortcoming in the perceptron design.
In addition to that distinction, although the signal strength summing into activation functions are parametrized, traditional ANNs, CNNs, and RNNs, are statically connected, something Intel claims they will correct with the Nirvana architecture in 2019 (which places into silicon that which we would call layer set up in Python or Java now.
There are at least three important biological neuron features that make the activation mechanism more than a function of a scalar input producing a scalar output, which renders questionable any algebraic comparison.
The decision about what activation function to use has largely been based on the analysis of convergence on a theoretical level combined with testing permutations to see which ones show the most desirable combinations of speed, accuracy, and reliability in ctheonvergence. By reliability is meant that convergence on the global optimum (not some local minimum of the error function) is reached at all for the majority of input cases.
This bifurcated research between the forks of practical machine learning and biological simulations and modeling. The two branches may rejoin at some point with the emergence of spiking - Accuracy - Reliability (completes) networks. The machine learning branch may borrow inspiration from the biological, such as the case of visual and auditory pathways in brains.
They have parallels and relationships that may be exploited to aid in progress along both forks, but gaining knowledge by comparing the shapes of activation functions is confounded by the above three differences, especially the temporal alignment factor and the entire timing of brain circuits which cannot be modeled using iterations. The brain is a true parallel computing architecture, not reliant on loops or even time sharing in the CPU and data buses.
The answer is We do not know. Odds are, we will not know for quite a while. The reason for this is we cannot understand the "code" of the human brain, nor can we simply feed it values and get results. This limits us to measuring currents of the input and output on test subjects, and we have had few such test subjects that are human. Thus, we know almost nothing about the human brain, including the activation function.
The human brain does have an activation function which is sigmoid like when considering the membrane potential at the Axon Initial Segment (AIS) as input and the rate of action potential as the output.
There is a threshold membrane potential under witch no action potential is generated, and there is a maximum output rate the neuron may produce due to the hyper polarisation after the action potential spike.
The biological neuron adjust the slope of the sigmoid and its mean (or the threshold if you like). The adjustment of the mean (and threshold) is quite fast as the AIS has a particular structure allowing to move the voltage sensitive sodium pores with an intracellular muscle (actin) and skeleton (microtubules).
.
From "The Axon Initial Segment: An Updated Viewpoint" Christophe Leterrier, Journal of Neuroscience 28 February 2018, 38 (9) 2135-2145; https://doi.org/10.1523/JNEUROSCI.1922-17.2018
The sigmoid change is normalizing the output. When the membrane potential is artificially strongly stimulated, the average output rate is reduced, and when the stimulation is reduced, the average output rate is increased.
This normalization makes sense with recursive neural network. The system must be maintained in a stable regime.
Note that the sigmoid is not symmetric and that the biological signal processing is stochastic. The action potential spike is a dirac which thus doesn't carry any information in itself. It is the relative time of spikes that carries the information.
My interpretation of the question was 'what activation function in an artificial neural network (ANN) is closest to that found in the brain?'
Whilst I agree with the selected answer above, that a single neuron outputs a dirac, if you think of a neuron in an ANN as modelling the output firing rate, rather than the current output, then I believe ReLU might be closest?
http://jackterwilliger.com/biological-neural-networks-part-i-spiking-neurons/
The basic neuron firing is similar to a spiking function. However, there is both an immediate (enough positives charges leads to overcoming activation threshold for the axon hillock, leading to more jumping in), and feedback effect. not every neuron “fires” the same way, as transcriptors can produce more channels to increase influx of charge and strengthen an activation. Axon terminal also have vesicles with neurotransmitters. Only a few hundred neurotransmitters exist, ex dopamine, similar to an alphabet which can be used to create different words or effects in different areas of the brain.
So while the basic concept can be analogous to spiking / Hebbian activation, in reality there is modulation of states which make it more complex.
