In computer vision is very common to use supervised tasks, where datasets have to be manually annotated by humans. Some examples are object classification (class labels), detection (bounding boxes) and segmentation (pixel-level masks). These datasets are essentially pairs of inputs-outputs which are used to train Convolutional Neural Networks to learn the mapping from inputs to outputs, via gradient descent optimization. But animals don't need anybody to show them bounding boxes or masks on top of things in order for them to learn to detect objects and make sense of the visual world around them. This leads me to think that brains must be performing some sort of self-supervision to train themselves to see.

What does current research say about the learning paradigm used by brains to achieve such an outstanding level of visual competence? Which tasks do brains use to train themselves to be so good at processing visual information and making sense of the visual world around them? Or said in other words: how does the brain manage to train its neural networks without having access to manually annotated datasets like ImageNet, COCO, etc. (i.e. what does the brain use as ground truth, what is the loss function the brain is optimizing)? Finally, can we apply these insights in computer vision?

Update: I posted a related question on Psychology & Neuroscience StackExchange, which I think complements the question I posted here: check it out

  • $\begingroup$ Cross-posted: psychology.stackexchange.com/q/21965/11209, ai.stackexchange.com/q/11666/1794, datascience.stackexchange.com/q/48645/8560. Please do not post the same question on multiple sites. Each community should have an honest shot at answering without anybody's time being wasted. $\endgroup$
    – D.W.
    Commented Apr 5, 2019 at 17:20
  • $\begingroup$ Your brain is not a computer (see this or this). Your brain doesn't optimize a loss function. Also, even if I reluctantly accept this analogy, I feel like the visual system has plenty of "external supervision" from the other senses, etc. $\endgroup$ Commented Apr 10, 2019 at 22:25
  • $\begingroup$ @sfmiller940 if the brain is not optimizing a loss, then what mechanism does the brain use to learn new things? Regarding other senses, can you elaborate more on how they exert supervision on the visual system? $\endgroup$ Commented Apr 11, 2019 at 3:17
  • $\begingroup$ @PabloMessina I don't think anybody knows exactly how the brain learns. But I'm not an expert, so I imagine that neuroscientists (or similar) would know much more about what is and isn't known. As far as supervision, let's consider the senses of touch and hearing. If we see a ball, we don't necessarily know if it's a discrete object but, if we can touch it, then we get additional information about it's shape and relation to other objects. Similarly we can hear sounds and voices that can instruct us directly or indirectly. Other senses seem a form of supervision that guides visual learning. $\endgroup$ Commented Apr 11, 2019 at 16:59
  • $\begingroup$ @sfmiller940 "Other senses seem a form of supervision that guides visual learning", probably, but how? $\endgroup$ Commented Apr 11, 2019 at 21:00

1 Answer 1


I think you are slightly confusing 2 problems. 1 being classification of meta visual elements and the other being the visual system itself.

Our visual system, when it comes to processing information, has had billions of years of iteration(training), so that at birth(and before), we are already tuned for the processing of visual stimuli, as well as have the mechanisms to decipher objects in our spatial field of view.

These two papers(L1, L2), have a great deal of information about the evolution of our visual system and its processing. The second speculates on the connection of said evolution and the construction of "seeing systems" very interesting.

For further inquiry on this in particular, check out David Marr. He was probably the most influential early computer vision mind. He still is mentioned in many top-down AGI and computer vision research projects to this day.

  • $\begingroup$ Do you mean babies are born right off the bat already identifying people, dogs, cats, mountains, water, intuitive physics, predicting object trajectories, etc.? Or do their brains learn this over time instead? If your answer to the last question is yes, then how do brains manage to find training examples to train their neural networks without having access to manually annotated datasets like ImageNet, COCO, etc.? $\endgroup$ Commented Apr 5, 2019 at 13:56
  • $\begingroup$ @PabloMessina while tests on infants are difficult for obvious reasons, yes. We're born with ingrained classification functionality and can identify spatial forms after and likely before we are out of the womb. Physics is evolutionary as well(at least for examples relevant to what our ancestors would of faced, lions, snakes, etc.). Some things get learned over time as well, like object permanence. Trying to compare our visual system to convolutional neural networks is a very rough analogy and usually is only used for instructive reasons. Our visual system is more likely than not convolutional. $\endgroup$ Commented Apr 5, 2019 at 14:09
  • $\begingroup$ Likely not convolutional* $\endgroup$ Commented Apr 7, 2019 at 6:03
  • $\begingroup$ Hey @hisairnessag3, your answer motivated me to ask this question in Psychology & Neuroscience, feel free to answer it if you want $\endgroup$ Commented Apr 7, 2019 at 14:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .