5
$\begingroup$

How can we prove that an auto-associator network will continue to perform if we zero the diagonal elements of a weight matrix that has been determined by the Hebb rule? In other words, suppose that the weight matrix is determined from $W = PP^T- QI$, where $Q$ is the number of prototype vectors.

I have been given a hint: show that the prototype vectors continue to be eigenvectors of the new weight matrix.

This is a question from Neural Network Design (2nd Edition) book by Martin T. Hagan, Howard B. Demuth, Mark H. Beale, Orlando De Jesus .

Resource : E7.5 p 224-225

$\endgroup$
3
  • $\begingroup$ I found this but I don't think it's what is needed for my question . $\endgroup$
    – estamos
    Dec 14, 2019 at 18:23
  • $\begingroup$ Some more theory about autoassociative nets can be found here p 48 . $\endgroup$
    – estamos
    Dec 14, 2019 at 18:32
  • $\begingroup$ In addition some relative paper p 16 - 20 $\endgroup$
    – estamos
    Dec 15, 2019 at 10:17

1 Answer 1

0
$\begingroup$

As we have an autoassociative network prototype vectors are both input and output vectors. So , we have that : \begin{equation} T = P \end{equation}

\begin{equation} W = PP^T - QI = TP^T - QI = \sum_{q=1}^Q p_q p_q^T - QI \end{equation}

Applying a prototype vector as input :

\begin{equation} \alpha = W \cdot p_k = \sum_{q=1}^Q p_k p_q p_q^T - QIp_k \end{equation}

Because they are orthogonal we have : \begin{equation} \alpha = p_k(p_k^T\cdot p_k) - Q \cdot I \cdot p_k = p_k (p_k^T\cdot p_k - Q \cdot I ) = p_k (\textbf{R - Q $\cdot$ I}) \end{equation}

where $R-Q\cdot I$ is the length of vectors

So since ,

\begin{equation} W \cdot p_k= (R - Q \cdot I) \cdot p _k \end{equation} prototype vectors continue to be eigenvectors of the new weight matrix .

It is often the case that for auto-associative nets, the diagonal weights (those which connect an input component to the corresponding output component) are set to 0. There are papers that say this helps learning. Setting these weights to zero may improve the net's ability to generalize or may increase the biological plausibility of the net. In addition, it is necessary if we use iterations (iterative nets) or the delta rule is used

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .