Can a vanilla neural network theoretically achieve the same performance as CNN?

Question

I perfectly understand that CNN takes into account the local dependency of each pixel to the nearby pixels. In addition, CNNs are spatially invariant which means that they are able to detect the same feature anywhere in the image. These qualities are useful in image classification problems given the nature of the problem.

How does a vanilla neural net exactly falls short on these properties? Am I right in claiming that a vanilla neural net has to learn a given feature in every part of the image? This is different than how a CNN does it, which learns the feature once and then detects it anywhere in the image.

How about local pixel dependancy? Why can't a vanilla neural network learn the local dependency by relating one pixel to its neighbors in the 1D input?

In other words, is there more information present while training a CNN that are simply absent when training a normal NN? Or is a CNN just better at optimizing in the space of image classification problems?

Answer 1

All CNNs can be represented as vanilla networks on the flattened image data. Just to do so, you would need A LOT of parameters (most of which would be 0) for what CNNs do freely. You can think of a CNN as reusing a filter on a masked input (whichever receptive field it's looking at whatever point during the convolution) repetitively.

In other words, fully connected layers use all the information, so it can still learn spatial dependence as a CNN does, while CNNs for each neuron only look at a specific receptive field and will reuse that filter for all neurons in that channel. This constraint saves computation and allows wider and deeper models under some budget.

This is nice because the hypothesis of why CNN's work are, is that at each point in the network we care about looking at localized features rather than global ones and that creating a composition of these makes it so even if each neuron only relates to a handful of neurons in the previous layer, the receptive field from the initial image can still be quite large if not the whole thing.

Take away: CNNs are an efficient implementation of a vanilla NN, given the locality constraint that each neuron only looks at a small localized subset of neurons from the previous layer.