Hopfield Nets are able to store a vector and retrieve it starting from a noisy version of it. They do so setting weights in order to minimise the energy function when all neurons are set equal the vector values, and retrieve the vector using the noisy version of it as input and allowing the net to settle to an energy minimum.

Leaving aside problems like the fact that there is no guarantee that the net will settle in the nearest minimum etc –problems eventually solved with Boltzmann machines and eventually with back-propagation– the breakthrough was they are a starting point for having abstract representations. Two versions of the same document would recall the same state, they would be represented, in the network, by the same state. As Hopfield himself wrote: "The present modeling might then be related to how an entity or Gestalt is remembered or categorized on the basis of inputs representing a collection of its features."

On the other side, the breakthrough of deep learning was the ability of building multiple, hierarchical representation of the input, eventually leading to make AI-practitioners' life easier, simplifying feature engineering. (see eg "Representation Learning: A Review and New Perspectives", Bengio, Courville, Vincent).

From a conceptual point of view, one can see deep learning as a generalisation of Hopfield nets: from one single representation to a hierarchy of representation. (I believe)

The question: is that true from a computational/topological point of view as well? Not considering how "simple" Hopfield networks were (2-state neurons, undirected, energy function), can one see each layer of a network as a Hopfield network and the whole process as a sequential extraction of previously memorised Gestalt, and a reorganisation of these Gestalt?


Deep Learning is not a generalization of Hopfield networks. Deep Learning is a "generalization" of the neural networks/connectionism field started by Rumelhart and McClelland.

There are two kinds of neural networks:

  • Directed (Perceptron, MLP, ConvNets, RNNs, etc.)
  • Undirected (Hopfield Nets, Boltzmann Machines, Energy-based models, etc.)

Any of these can be made deep. As you said, Boltzmann machines are the probabilistic version of Hopfield Networks, and there has been a lot more work on deepifying these models than Hopfield nets: Deep Boltzmann machines, Deep Belief Networks, and deep energy models. Hinton is really the guy you want to read to learn about these models, but you can have a look at this paper which compares the three models.

Not sure about the Gestalt organisation. I guess I'll leave that to your interpretation.

  • $\begingroup$ Not very informative or doesn't cite any sources for the conclusions. The OP of the question clearly knows about DL and Hopfield Nets. $\endgroup$ – DuttaA Apr 27 '19 at 7:39
  • $\begingroup$ sorry for the late comment... thanks. I know DL is not considered a generalization of Hopfield networks. My question –probably not clear enough– was about the emergence of the ability of categorise (Gestalt) in NN. $\endgroup$ – Mario Alemi Jul 2 '19 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.