What can be deduced about the "algorithm" of backpropagation/gradient descent?

Question

On this video

Link to video

a neurologist starts by saying that we do not know how neurons calculate gradients for backpropagation.

At minute 30:39 hes showing faster convergence for "our algorithm", which seems to converge faster than backpropagation.

After 34:36 it goes explaining how "neurons" in the brain are actually packs of neurons.

I do not really understand all that he says, so I infer that those packs of neurons (which seem depicted as a single layer) are the ones who calculate the gradient. It would make sense if each neuron makes a sightly different calculation, and then each other communicate the difference in results. That would allow to deduce a gradient.

What can be deduced, from the presented information, about the purported "algorithm"?? (From the viewpoint of improving convergence of an artificial neural network).

Answer 1

There are 3 separate issues that are often confounded in Deep Learning and Neuroscience:

1. Deep Learning is inspired by the way the biological brain works.
2. Deep Learning is how the biological brain works.
3. Deep Learning can model how the biological brain works.

Number 1 is accurate. The brain has many layers and many connections. Those principles have informed Deep Learning models.

Number 2 has little evidence to support that claim. The biological brain learns at the cellular level in very different ways than how any Deep Learnings system learns.

Number 3 is a current topic of research. Deep Learning is very good at learning patterns. There are good reasons to believe that Deep Learning can learn patterns in the brain. However, those Deep Learning models will not automatically give insight into the biological processes of the brain.

The video is an example of #1. Inspired by our current understanding of neurobiology, let's build better Deep Learning algorithms. These new algorithms might perform better on machine learning benchmarks. However, these algorithms are not better models of the biological brain. In order to understand the algorithms, the language of biology might not be helpful. It might be better to describe them mathematically.