If neurons performed the operation of an entire layer, would that make the neural network more effective?

Question

(I have a very primitive understanding of neural networks, so please forgive the lack of technicality here.)

I am used to seeing a neuron in a neural network as something that-

1. Takes the inputs and multiplies them by their weights,
2. then sums them up,
3. and after that it applies the activation function to the sum.

Now, what if it was "smarter"? Say, a single neuron could do the function of an entire layer in a network, could that make the network more effective? This comes from an article I was reading at Quanta, where the author says:

Later, Mel and several colleagues looked more closely at how the cell might be managing multiple inputs within its individual dendrites. What they found surprised them: The dendrites generated local spikes, had their own nonlinear input-output curves and had their own activation thresholds, distinct from those of the neuron as a whole. The dendrites themselves could act as AND gates, or as a host of other computing devices.

...realised that this meant that they could conceive of a single neuron as a two-layer network. The dendrites would serve as nonlinear computing subunits, collecting inputs and spitting out intermediate outputs. Those signals would then get combined in the cell body, which would determine how the neuron as a whole would respond.

My thoughts: I know that Backpropagation is used to "teach" the network in the normal case, and the fact that neurons are simply activation buttons is somehow related to that. So, if neurons were to be more complicated, it would reduce efficiency. However, I am not sure of this: why would complex individual components make the network less effective?

Answer 1

Say, a single neuron could do the function of an entire layer in a network, could that make the network more effective?

That depends what you mean by "more effective". In terms of number of neurons to achieve the same result, then you should need fewer units. In terms of being able to calculate an end result for any specific problem, then no, because you can generally solve a problem using simpler units by adding more of them.

If this could somehow be done using less resources, it might reduce overall costs. In a biological system, there are possibly overheads per cell in order to maintain it on top of the costs for calculation, so it may be better to do more than the simplest calculation in each cell. Further to that, there may be an optimal amount of processing that each cell could do (this is all conjecture on my part).

In an artifical neural network, the calculations are the only thing being considered, there is no separate overhead per neuron.

There are neural network architectures with complex "sub-units". Probably the most well known are the recurrent neural network designs for LSTM and gated recurrent units (GRU), plus "skip connections" in residual neural networks. For efficiency these are normally processed in groups per layer with matrix processing functions, but you can also view them as per-neuron complexities.

I am not sure of this- why would complex individual components make the network less effective?

If the complexity was not used, or not really needed, in some of the units, then it would be wasted capacity. In a biological system, this might correspond to maintaining cells larger than they needed to be. In an artificial system, it would mean using memory and CPU to calculate interim values that were not needed for the task at hand.