Hopfield networks are able to store a vector and retrieve it starting from a noisy version of it. They do so setting weights in order to minimize the energy function when all neurons are set equal to the vector values, and retrieve the vector using the noisy version of it as input and allowing the net to settle to an energy minimum.

Leaving aside problems like the fact that there is no guarantee that the net will settle in the nearest minimum etc – problems eventually solved with Boltzmann machines and eventually with back-propagation – the breakthrough was they are a starting point for having abstract representations. Two versions of the same document would recall the same state, they would be represented, in the network, by the same state.

As Hopfield himself wrote in his 1982 paper Neural networks and physical systems with emergent collective computational abilities

The present modeling might then be related to how an entity or Gestalt is remembered or categorized on the basis of inputs representing a collection of its features.

On the other side, the breakthrough of deep learning was the ability to build multiple, hierarchical representations of the input, eventually leading to making AI-practitioners' life easier, simplifying feature engineering. (see e.g. Representation Learning: A Review and New Perspectives, Bengio, Courville, Vincent).

From a conceptual point of view, I believe one can see deep learning as a generalization of Hopfield nets: from one single representation to a hierarchy of representation.

Is that true from a computational/topological point of view as well? Not considering how "simple" Hopfield networks were (2-state neurons, undirected, energy function), can one see each layer of a network as a Hopfield network and the whole process as a sequential extraction of previously memorized Gestalt, and a reorganization of these Gestalt?


Deep Learning is not a generalization of Hopfield networks. Deep Learning is a "generalization" of the neural networks/connectionism field started by Rumelhart and McClelland.

There are two kinds of neural networks:

  • Directed (Perceptron, MLP, ConvNets, RNNs, etc.)
  • Undirected (Hopfield Nets, Boltzmann Machines, Energy-based models, etc.)

Any of these can be made deep. As you said, Boltzmann machines are the probabilistic version of Hopfield Networks, and there has been a lot more work on deepifying these models than Hopfield nets: Deep Boltzmann machines, Deep Belief Networks, and deep energy models. Hinton is really the guy you want to read to learn about these models, but you can have a look at this paper which compares the three models.

Not sure about the Gestalt organisation. I guess I'll leave that to your interpretation.

| improve this answer | |
  • $\begingroup$ My question wasn't probably clear enough. I was asking about the emergence of the ability to categorise (Gestalt) in NN. $\endgroup$ – Mario Alemi Jul 2 '19 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.