3
$\begingroup$

I am looking for a rigorous mathematical proof for finding the several local minima of the Hopfield networks. I am searching for something rigorous, a demonstration, not just letting the network keep updating its neurons and wait for noticing a stable state of the network.

I have looked virtually everywhere, but I found nothing.

Is there a rigorous proof for Hopfield minima? Could you give me ideas or references?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
3
$\begingroup$

See the paper On the Convergence Properties of the Hopfield Model (1990), by Jehoshua Bruck.

In the first section of the paper, J. Bruck describes the Hopfield network (popularized by J. J. Hopfield in 1982 in his paper Neural networks and physical systems with emergent collective computational abilities, hence the name of the network), then he describes the notation that is used throughout the paper and he gives some examples where a simple Hopfield network (with two nodes) converges to stable states (of the network) and cycles (of which the author also gives a definition).

The usual proofs of the convergence properties of Hopfield networks involve the concept of an energy function, but, in this paper, J. Bruck uses an approach (based on an equivalent formulation of the Hopfield network as an undirected graph) that does not involve an energy function, and he unifies three apparently different convergence properties related to Hopfield networks (described in part $C$ of the section Introduction). More specifically, finding the global maximum of the energy function associated with the Hopfield network operating in a serial mode (which is defined in part $A$ of the Introduction section of the paper) is equivalent to find a minimum cut in the undirected graph associated with this Hopfield network.

Furthermore, note that the proofs of convergence of the Hopfield networks actually depend on the structure of the network (more specifically, its weight matrix). For example, if the weight matrix $W \in \mathbb{R}^{n \times n}$ (where $n$ is the number of nodes in the network) associated with the Hopfield network is a symmetric matrix with the elements of the diagonal being non-negative, then the network will always converge to a stable state.

See also the chapter 13 The Hopfield Model of the book Neural Networks - A Systematic Introduction (1996) by Raul Rojas.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thank you for your answer. Sorry I if was not clear but I intend my question the way I asked it. I understand why Hopfield net (the proof) converge to a local minima, a stable state etc... Precisely I already read the resources that you mentionned and they only deal with that issue (I found Rojas book very useful thou!) $\endgroup$
    – lolodino77
    Jul 22, 2019 at 21:51
  • $\begingroup$ @ladangvu Ok, so what's your question? What do you want more than this? The resources I provided are already quite rigorous, so I really don't get your question. $\endgroup$
    – nbro
    Jul 22, 2019 at 21:53
  • $\begingroup$ What I am looking for is a demonstration for finding all these stable states. Of course everybody states that they are the patterns learned, their opposite (negative counterpart) and spurious states. But nowhere did I find proof that the patterns learned are local minima of the energy function. $\endgroup$
    – lolodino77
    Jul 22, 2019 at 21:55
  • $\begingroup$ @ladangvu What do you mean by "demonstration for finding all these stable states"? Have you read the mentioned paper "On the Convergence Properties of the Hopfield Model"? Even in the initial examples, the author gives examples of states that are the local minimum of the energy function (that is, where the Hopfield network reaches a stable state)? Also, have you read the part of my answer where I say "note that the proofs of convergence of the Hopfield networks actually depend on the structure of the network". $\endgroup$
    – nbro
    Jul 22, 2019 at 21:58
  • $\begingroup$ A mathematical demonstration (as we regularly do in multivariable functions optimization) for determining the local minimizer and minimum of the energy function E. A demonstration like any function that we would like to minimize, for example from my course : f(x,y) = x^3+y^3-3cxy where c belongs to IR*. After calculate the partial derivatives, finding the critical points and and calculate the determinant of the hessian at these points, we found that : (0,0) is a saddle point and (c,c) a local maximizer if c < 0 or a local minimizer if c > 0. $\endgroup$
    – lolodino77
    Jul 22, 2019 at 22:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .