What makes learned feature detectors specialize in CNN?

Question

It has been shown that it is possible to use unsupervised learning techniques to produce good feature detectors in CNNs. I can't understand what drives specialization of those feature detectors. In publications https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf (page 6) they show a very reasonable set of edge and blob detectors with little or no overlap. It goes against intuition - in absence of incentive to specialize, one would expect at least some duplication of learned kernels.

Answer 1

You are thinking along the correct line when you consider intuitivity when examining the strange self-specialization aspect of the layers in any deep network that has been tuned, through toil, mental machinations, and late nights, to work.

Do AI experimenters manipulated parameters, topology, and algorithmic hierarchies to force circuit evolution that produces data flows through specialized functions like this one.

• Noisy, normalized, cubes (horizontal, vertical, depth) of integers straight from video hardware or the video channel of a multimedia file
• Indications of edges
• Indications of corners, ends, and bends
• Indications of relative angles (in the case of size independent recognition)
• Indications of shapes
• Indications of 2D topology
• Indications of object forms
• Indication of scene
• Indication of action (when a hypercube is used and frame is another dimension)

Of course, one could design a solution and then tune layers, one by one, to follow the design, but it is not necessary because of the concept of entropy removal, the elimination of information that is redundant or irrelevant.

The principle in information science is this, in general: As redundancy is removed from the information, the number of bits representing the information decreases and the level of abstraction increases.

In minimalist art, a single word an identical twin might say to the other twin as the train rolls away, or a good Pictionary player, given a small window of information passing opportunity, adaptations can be made to pass large amounts of information base on previously agreed upon conventions.

When you think of it this way, it makes sense that by narrowing the bit width of the aggregate data representation from layer to layer that only certain parametric optimizations will provide an end-to-end indication of convergence, given a goal such as fully categorizing objects based on a small number of features.

When Leibniz envisioned a world tied together by mathematical certainty, he was overly optimistic, however he was correct in at least one important respect, mathematics had a reality all its own. The success of Newtonian mechanics is one early example of a huge reality that has become apparent and has been so widely applicable, deduced from a thought experiment involving a cannon ball and the moon.

There are few demonstrations of discoveries that originally appear to be wild inventions than the amazingly accurate prediction of the previously anomalous orbit of Mercury. But Einstein was not surprised by his success. He had deduced general relativity from the challenge of Ernst Mach and the undeniable results from light and gravity experiments. He discovered how it must be before it was found.

In the same sense, the above sequence of events can be proven mathematically to be the efficient strategy for vision. The sequence of processes to transform incoming visual signals to collision avoidance control was there in mathematics before there were visual receptors on microbes. That is why the narrowing of the visual pathways of a Pterodactyl have similarities to those of a shark shark even though their common ancestor may not have been able to see.

The incentive to specialize is the incentive to converge on the objective even after redundancy is stripped by force by limiting the information channel aperture. Even if the aperture is in multiple dimensions, the same principle applies.