I perfectly understand that CNN takes into account the local dependency of each pixel to the nearby pixels. In addition, CNNs are spatially invariant which means that they are able to detect the same feature anywhere in the image. These qualities are useful in image classification problems given the nature of the problem.
How does a vanilla neural net exactly falls short on these properties? Am I right in claiming that a vanilla neural net has to learn a given feature in every part of the image? This is different than how a CNN does it, which learns the feature once and then detects it anywhere in the image.
How about local pixel dependancy? Why can't a vanilla neural network learn the local dependency by relating one pixel to its neighbors in the 1D input?
In other words, is there more information present while training a CNN that are simply absent when training a normal NN? Or is a CNN just better at optimizing in the space of image classification problems?