4
$\begingroup$

I am familiar with the currently popular neural network models that have weights and are trained with backpropagation and gradient descent.

However, I came across a different type of neural network popular in the 1980s and 1990s. The Hopfield network is one of the most classic. These neural network models different in the following ways:

  1. They do not have parameter weights and bias to train or to learn from data.
  2. They used a circuit diagram to present the model.
  3. The model can be simplified as an ODE system and has a Lyapunov function as objective.

Please take a look at these two papers in the 1980s:

  1. Neurons with graded response have computational properties like those of two state neurons (J.J. Hopfield)

  2. Neural Networks for Non-linear Programming (M.P Kennedy & L.O Chua)

$\endgroup$
4
  • $\begingroup$ You seem to be asking us why John Hopfield used a circuit diagram when he wrote his paper describing his neural networks. Given that we're not able to read Hopfield's mind, I don't think it's possible to answer this question (aside from giving the obvious answer, which is "because Hopfield thought that was a good way to describe his neural networks"). $\endgroup$ Apr 29 at 21:29
  • $\begingroup$ I agree with the previous comment. I don't really understand what your question really is here. Can you clarify that? Moreover, in the body, you only talk about Hopfield networks, while in the title you're talking about "some neural networks". Which ones, apart from the Hopfield network? In other words, if you're just referring to the Hopfield network, don't use "some", but be as much specific as possible. $\endgroup$
    – nbro
    Apr 30 at 10:56
  • $\begingroup$ 1. I don't agree with the user 'nbro' comment. I believe I have stated the question clearly. I am asking about a general type of neural network models(many) presented by circuit diagram, and I referred to the Hopfield network(one) as a classic example. The Hopfield network(one) obviously belongs to the general type of neural network models(many) I was talking about. I don't see any confusion about this. $\endgroup$
    – dawen
    Apr 30 at 12:31
  • $\begingroup$ 2. I don't agree with the user ' Tanner Swett' comment. It is possible to answer the question. You don't need to go inside someone brain to understand his thought. There must be some reason or some benefit to using the circuit diagram, the reason may state somewhere I don't know. And this is why I am asking the question here, hope someone knows it can give an answer. $\endgroup$
    – dawen
    Apr 30 at 12:38
3
$\begingroup$

In the early days of neural networks the theorists and practitioners were educated in mathematics, psychology, neurophysiology, electrical engineering, and neurobiology. Computer science was still in its infancy. The first neural networks were modeled as electrical circuits.

There is evidence of this in the 1943 paper by Warren McCulloch and Walter Pitts [1], and a 1956 paper by Rochester et al. [2].

The latter paper uses terms such as 'circuits' and 'switching'. One idea in the paper is explained in terms of a "Eccles-Jordan Flip Flop circuit" although there are no drawings. Nathanial Rochester had designed the IBM 701k [3] and "led the first effort to simulate a neural network" [4]

Brain structure was discussed in terms of 'neural circuits' as early as 1937 [5].

I am not sure when the first electrical circuit diagram appeared in publication, but it makes sense that early neural network designers, would have thought of their implementation as such.

References:

$\endgroup$
2
$\begingroup$

Well, the goal of any paper is to allow the reader to understand what the author is trying to describe.

A lot of people have a lot of experience looking at circuit diagrams and figuring out those circuits will do. For these people, a circuit diagram may be the clearest and easiest way for them to understand how a particular thing works. So, it makes sense that an author would include a circuit diagram, in order to make it easy for those people to understand the concepts.

There are two particular reasons why a circuit diagram is especially likely to show up in a paper about neural networks:

The first reason is that analog circuits are closely related to ordinary differential equations, and digital circuits are closely related to sequential logic. So, if you have a neural network or something that uses ordinary differential equations or sequential logic, then a circuit diagram might be a simple way to express how it works.

The second reason is that a lot of researchers who are familiar with computers are also familiar with electronic circuits. This was especially true in the early days of computers, when people had to be familiar with electronics, math, or both in order to understand how to program a computer.

$\endgroup$
2
  • $\begingroup$ Thanks for your answer. Can you give more detail or explanation or reference that circuit diagrams are closely related ODE system? $\endgroup$
    – dawen
    Apr 30 at 17:02
  • $\begingroup$ @dawen That's the material that would be covered in an "introduction to electronic circuits" course. $\endgroup$ Apr 30 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.