TL;DR
If we buy into the idea visual cortex functions like a convolutional neural network, then there's a problem makes me scratch my head: how does brain force weight sharing as in convolutional network?
Okay, explain more
Obviously, there's no way for left visual cortex to directly tell the right visual cortex "hey, I've learned some new stuff, copy me!!" (or is there?). Then, if the learned features are diverse across visual field, how does it keep the translation invariance property?
For example, you already know English characters, you can recognize them with your both eyes. Now that you wanna learn some Chinese and you excercise your right brain at the same time, so you closed your right eye and memorized a new character. After that, certainly you can recognize the new character with solely your right eye. But why?
The answer may be, the object / higher-level feature detection happens in a higher level cortex, which receives entire visual field. There may be also some transfer/one-shot learning taking place. But then, if a newborn baby trying to learn the low level visual features, he/she would definitely face the weight sharing problems.
A possible explanation would be, the baby will be exposed to very large amount of data and eventually learn invariance. Large amount of data reduces overfitting but doesn't guarantee deterministic convergence. If we train the same CNN model on the same dataset, however using different random generator seeds, there's a big chance the same feature detector will appear in a different channel, or a difference set of features appear as linear recombination.
If there's no way to share weights, the brain would learn a lot different feature combinations across the entire visual field, how does it still able to consistently solve visual invariance problem?