How can we prove that an autoassociator network will continue to perform if we zero the diagonal elements of a weight matrix?

Question

How can we prove that an auto-associator network will continue to perform if we zero the diagonal elements of a weight matrix that has been determined by the Hebb rule? In other words, suppose that the weight matrix is determined from $W = PP^T- QI$, where $Q$ is the number of prototype vectors.

I have been given a hint: show that the prototype vectors continue to be eigenvectors of the new weight matrix.

This is a question from Neural Network Design (2nd Edition) book by Martin T. Hagan, Howard B. Demuth, Mark H. Beale, Orlando De Jesus .

Resource : E7.5 p 224-225

Answer 1

As we have an autoassociative network prototype vectors are both input and output vectors. So , we have that : \begin{equation} T = P \end{equation}

\begin{equation} W = PP^T - QI = TP^T - QI = \sum_{q=1}^Q p_q p_q^T - QI \end{equation}

Applying a prototype vector as input :

\begin{equation} \alpha = W \cdot p_k = \sum_{q=1}^Q p_k p_q p_q^T - QIp_k \end{equation}

Because they are orthogonal we have : \begin{equation} \alpha = p_k(p_k^T\cdot p_k) - Q \cdot I \cdot p_k = p_k (p_k^T\cdot p_k - Q \cdot I ) = p_k (\textbf{R - Q $\cdot$ I}) \end{equation}

where $R-Q\cdot I$ is the length of vectors

So since ,

\begin{equation} W \cdot p_k= (R - Q \cdot I) \cdot p _k \end{equation} prototype vectors continue to be eigenvectors of the new weight matrix .

It is often the case that for auto-associative nets, the diagonal weights (those which connect an input component to the corresponding output component) are set to 0. There are papers that say this helps learning. Setting these weights to zero may improve the net's ability to generalize or may increase the biological plausibility of the net. In addition, it is necessary if we use iterations (iterative nets) or the delta rule is used