How can we prove that an auto-associator network will continue to perform if we zero the diagonal elements of a weight matrix that has been determined by the Hebb rule? In other words, suppose that the weight matrix is determined from $W = PP^T- QI$, where $Q$ is the number of prototype vectors.

I have been given a hint: show that the prototype vectors continue to be eigenvectors of the new weight matrix.

This is a question from Neural Network Design (2nd Edition) book by Martin T. Hagan, Howard B. Demuth, Mark H. Beale, Orlando De Jesus .

Resource : E7.5 p 224-225

  • $\begingroup$ Hi and welcome to this community! Is this a question from a book or something similar? If yes, please, edit your post to include a link to such a resource. To be more precise, when you say "auto-associator network", are we talking about Hopfield networks or another neural network? $\endgroup$ – nbro Dec 14 '19 at 14:39
  • 1
    $\begingroup$ @nbro I just edited and added the resource . About autoassociator network i also have the same doubt but i can't really understand what the question means . Any idea is welcome . $\endgroup$ – estamos Dec 14 '19 at 15:26
  • $\begingroup$ I don't remember exactly but Raul Rojas book on Neural Nets (free) has some treatment on this similar topic. Although I don't know whether it is exactly your question or not. $\endgroup$ – DuttaA Dec 14 '19 at 17:37
  • 1
    $\begingroup$ You can also try DataScience.SE. $\endgroup$ – DuttaA Dec 23 '19 at 12:20
  • 1
    $\begingroup$ Also there are sites like CrossValidated.SE. I don't know your research topic but from my experience if it isn't quite well known or broad topic there are slim chances of getting a satisfactory answer. $\endgroup$ – DuttaA Dec 23 '19 at 12:36

As we have an autoassociative network prototype vectors are both input and output vectors. So , we have that : \begin{equation} T = P \end{equation}

\begin{equation} W = PP^T - QI = TP^T - QI = \sum_{q=1}^Q p_q p_q^T - QI \end{equation}

Applying a prototype vector as input :

\begin{equation} \alpha = W \cdot p_k = \sum_{q=1}^Q p_k p_q p_q^T - QIp_k \end{equation}

Because they are orthogonal we have : \begin{equation} \alpha = p_k(p_k^T\cdot p_k) - Q \cdot I \cdot p_k = p_k (p_k^T\cdot p_k - Q \cdot I ) = p_k (\textbf{R - Q $\cdot$ I}) \end{equation}

where $R-Q\cdot I$ is the length of vectors

So since ,

\begin{equation} W \cdot p_k= (R - Q \cdot I) \cdot p _k \end{equation} prototype vectors continue to be eigenvectors of the new weight matrix .

It is often the case that for auto-associative nets, the diagonal weights (those which connect an input component to the corresponding output component) are set to 0. There are papers that say this helps learning. Setting these weights to zero may improve the net's ability to generalize or may increase the biological plausibility of the net. In addition, it is necessary if we use iterations (iterative nets) or the delta rule is used

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.