How does visual cortex share convolution weight

Question

TL;DR

If we buy into the idea visual cortex functions like a convolutional neural network, then there's a problem makes me scratch my head: how does brain force weight sharing as in convolutional network?

Okay, explain more

Obviously, there's no way for left visual cortex to directly tell the right visual cortex "hey, I've learned some new stuff, copy me!!" (or is there?). Then, if the learned features are diverse across visual field, how does it keep the translation invariance property?

For example, you already know English characters, you can recognize them with your both eyes. Now that you wanna learn some Chinese and you excercise your right brain at the same time, so you closed your right eye and memorized a new character. After that, certainly you can recognize the new character with solely your right eye. But why?

The answer may be, the object / higher-level feature detection happens in a higher level cortex, which receives entire visual field. There may be also some transfer/one-shot learning taking place. But then, if a newborn baby trying to learn the low level visual features, he/she would definitely face the weight sharing problems.

A possible explanation would be, the baby will be exposed to very large amount of data and eventually learn invariance. Large amount of data reduces overfitting but doesn't guarantee deterministic convergence. If we train the same CNN model on the same dataset, however using different random generator seeds, there's a big chance the same feature detector will appear in a different channel, or a difference set of features appear as linear recombination.

If there's no way to share weights, the brain would learn a lot different feature combinations across the entire visual field, how does it still able to consistently solve visual invariance problem?

Answer 1

In the human brain every common pattern is recognised by a multitude of pattern recognisers (be they neurons or microcolumns). That's pretty obvious because neurons die all the time, but we don't wake up and have forgotten how to recognise the letter 'A'. In fact we have to take out rather big chunks of the neocortex until functionality degrades, which is why Alzheimer's patients show symptoms only when the brain is already visibly messed up.

Translational invariance within a level of the hierarchy has to be learned. Basically you need filters for the same edge all over your V1. An object moving across your field of vision only becomes invariant in it's representation on a higher level. Unfortunately we don't magically learn a representation that we can easily turn and twist, our ability to recognise objects from an unusual angle degrades with how uncommon the angle is.

A nice thought experiment to illustrate the point: Imagine a cube sitting in front of you. Now pull one of the corners of the cube upwards until the cube is dangling in front of you, one corner pointing straight to the ceiling, the opposite corner pointing to the ground. Now indicate with your finger where the rest of the corner are.

If you are anything like me, this is really difficult. I think the first time I did't even realise I had to point out six corners!

Of course this is 3D and we might still have inbuilt 2D invariance, but it also turns out that faces that are turned upside down have to be processed much higher into the cortex to be recognised as faces and so on ...

Concerning the learning of a lot of different feature vectors across the field of vision: This might be prevented by the fact that the neocortex learns sequences of input by predicting (via depolarisation) the next pattern. So you might have the situation that a higher level tells the lower level that there is an object moving from left to right and the lower level will predict that the edge it is detecting at point y will reoccur at point y+x.

This setup differs from the training of NNs in two ways: The data is already translational and the translation is predicted, which facilitates learning the same features. Basically two pattern recogniser close to each other get pretty much the same pattern one after the other whenever something moves in a certain direction and the second occurrence is predicted, which means it will be more likely detected, which means it will be learned. (I don't want to dole out a lecture about the cortex, but the prediction is a big deal because it allows neurons to fire quicker which means they beat other neurons before they are laterally inhibited and only if they actually fire will they learn.)

As a disclaimer I want to add that this is just my current understanding of the issue and I'm pretty sure the actual explanation is more complicated. For example I've read that one of the layers of the cortex is important for translational invariance and this layer only exists in the levels close to the sensory input. It is conjectured that this layer (L4) doesn't do the sequence prediction stuff, so maybe having the same kind of input is enough or learning different feature vectors is not actually a problem. There is also the complicating issue that there is poorly understood interplay between the different layers and different levels of the cortex. I would recommend to ask a neuroscientist, except I don't think they figured this out yet.