13
$\begingroup$

Does the human brain use a specific activation function? I've tried doing some research, and as it's a treshold for whether the signal is sent through a neuron or not, it sounds a lot like ReLU. However, I can't find a single article confirming this. Or is it more like a step function (it sends 1 if it's above the treshold, instead of the input value).

$\endgroup$
  • 3
    $\begingroup$ Wlecome to ai.se...Hardly anything is known about the human brain as a whole..But the neurons individually communicate through spikes of current $\endgroup$ – DuttaA Apr 18 '18 at 11:06
10
$\begingroup$

The thing you were reading about is known as the action potential. It is a mechanism that governs how information flows within a neuron.

It works like this: Neurons have an electrical potential, which is a voltage difference inside and outside the cell. They also have a default resting potential, and an activation potential. The neuron tends to move towards the resting potential if it is left alone, but incoming electric activations from dendrites can shift its electric potential.

If the neuron reaches a certain threshold in electric potential (the activation potential), the entire neuron and its connecting axons goes through a chain reaction of ionic exchange inside/outside the cell that results in a "wave of propagation" through the axon.

TL;DR: Once a neuron reaches a certain activation potential, it electrically discharges. But if the electric potential of the neuron doesn't reach that value then the neuron does not activate.

Does the human brain use a specific activation function?

IIRC neurons in different parts of the brain behave a bit differently, and the way this question is phrased sounds as if you are asking if there is a specific implementation of neuronal activation (as opposed to us modelling it).

But in general behave relatively similar to each other (Neurons communicate with each other via neurochemicals, information propagates inside a neuron via a mechanism known as the action potential...) But the details and the differences they cause could be significant.

There are various biological neuron models, but the Hodgkin-Huxley Model is the most notable.

Also note that a general description of neurons don't give you a general description of neuronal dynamics a la cognition (understanding a tree doesn't give you complete understanding of a forest)

But, the method of which information propagates inside a neuron is in general quite well understood as sodium / potassium ionic exchange.

It (activation potential) sounds a lot like ReLU...

It's only like ReLU in the sense that they require a threshold before anything happens. But ReLU can have variable output while neurons are all-or-nothing.

Also ReLU (and other activation functions in general) are differentiable with respect to input space. This is very important for backprop.

This is a ReLU function, with the X-axis being input value and Y-axis being output value. enter image description here

And this is the action potential with the X-axis being time, and Y being output value.enter image description here

$\endgroup$
  • $\begingroup$ It is not a step function...Rather a Delta Dirac function..Slight miswording there $\endgroup$ – DuttaA Apr 18 '18 at 15:01
  • 2
    $\begingroup$ It's a step function if the x-axis is the input, but a Delta Dirac if the x-axis is time, you mean? $\endgroup$ – mlman Apr 20 '18 at 6:55
  • 1
    $\begingroup$ You are right but the point i'm trying to get at (about it being continuous and moving back to resting potential), ill upload a picture to make myself more clear $\endgroup$ – k.c. sayz 'k.c sayz' May 4 '18 at 21:18
2
$\begingroup$

The brains of mammals do not use an activation function. Only machine learning designs based on the perceptron multiply the vector of outputs from a prior layer by a parameter matrix and pass the result statelessly into a mathematical function.

Although the spike aggregation behavior has been partly modeled, and in far more detail than the 1952 Hodgkin and Huxley model, all the models require statefulness to functionally approximate biological neurons. RNNs and their derivatives are an attempt to correct that shortcoming in the perceptron design.

In addition to that distinction, although the signal strength summing into activation functions are parametrized, traditional ANNs, CNNs, and RNNs, are statically connected, something Intel claims they will correct with the Nirvana architecture in 2019 (which places into silicon that which we would call layer set up in Python or Java now.

There are at least three important biological neuron features that make the activation mechanism more than a function of a scalar input producing a scalar output, which renders questionable any algebraic comparison.

  • State held as neuroplastic (changing) connectivity, and this is not just how many neurons in a layer but also the direction of signal propagation in three dimensions and the topology of the network, which is organized, but chaotically so
  • The state held within the cytoplasm and its organelles, which is only partly understood as of 2018
  • The fact that there is a temporal alignment factor, that pulses through a biological circuit may arrive via synapses in such a way that they aggregate but the peaks of the pulses are not coincident in time, so the activation probability is not as high as if they were temporally aligned.

The decision about what activation function to use has largely been based on the analysis of convergence on a theoretical level combined with testing permutations to see which ones show the most desirable combinations of speed, accuracy, and reliability in ctheonvergence. By reliability is meant that convergence on the global optimum (not some local minimum of the error function) is reached at all for the majority of input cases.

This bifurcated research between the forks of practical machine learning and biological simulations and modeling. The two branches may rejoin at some point with the emergence of spiking - Accuracy - Reliability (completes) networks. The machine learning branch may borrow inspiration from the biological, such as the case of visual and auditory pathways in brains.

They have parallels and relationships that may be exploited to aid in progress along both forks, but gaining knowledge by comparing the shapes of activation functions is confounded by the above three differences, especially the temporal alignment factor and the entire timing of brain circuits which cannot be modeled using iterations. The brain is a true parallel computing architecture, not reliant on loops or even time sharing in the CPU and data buses.

$\endgroup$
1
$\begingroup$

The answer is We do not know. Odds are, we will not know for quite a while. The reason for this is we cannot understand the "code" of the human brain, nor can we simply feed it values and get results. This limits us to measuring currents of the input and output on test subjects, and we have had few such test subjects that are human. Thus, we know almost nothing about the human brain, including the activation function.

$\endgroup$
0
$\begingroup$

My interpretation of the question was 'what activation function in an artificial neural network (ANN) is closest to that found in the brain?'

Whilst I agree with the selected answer above, that a single neuron outputs a dirac, if you think of a neuron in an ANN as modelling the output firing rate, rather than the current output, then I believe ReLU might be closest?

http://jackterwilliger.com/biological-neural-networks-part-i-spiking-neurons/

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.