Hopfield networks are able to store a vector and retrieve it starting from a noisy version of it. They do so setting weights in order to minimize the energy function when all neurons are set equal to the vector values, and retrieve the vector using the noisy version of it as input and allowing the net to settle to an energy minimum.
Leaving aside problems like the fact that there is no guarantee that the net will settle in the nearest minimum etc – problems eventually solved with Boltzmann machines and eventually with back-propagation – the breakthrough was they are a starting point for having abstract representations. Two versions of the same document would recall the same state, they would be represented, in the network, by the same state.
As Hopfield himself wrote in his 1982 paper Neural networks and physical systems with emergent collective computational abilities
The present modeling might then be related to how an entity or Gestalt is remembered or categorized on the basis of inputs representing a collection of its features.
On the other side, the breakthrough of deep learning was the ability to build multiple, hierarchical representations of the input, eventually leading to making AI-practitioners' life easier, simplifying feature engineering. (see e.g. Representation Learning: A Review and New Perspectives, Bengio, Courville, Vincent).
From a conceptual point of view, I believe one can see deep learning as a generalization of Hopfield nets: from one single representation to a hierarchy of representation.
Is that true from a computational/topological point of view as well? Not considering how "simple" Hopfield networks were (2-state neurons, undirected, energy function), can one see each layer of a network as a Hopfield network and the whole process as a sequential extraction of previously memorized Gestalt, and a reorganization of these Gestalt?
