I'm reading papers regarding soft orthogonal regularization, $\frac \lambda 4||WW^\intercal - I||_F^2$, over a deep neural network whose activation function is ReLU and weight matrix $W$ is initialized orthogonal.

To my understanding, the orthogonal penalty's purpose is to encourage $W$ to become more orthogonalized with each update. But the derived total gradient would be $\nabla = \lambda W(WW^\intercal - I) + \nabla H$, for $H$ be the loss function. This suggests that after the first update, there is no penalty for orthogonality, and thus it is unlikely that $W$ will maintain its orthogonality.

The ability to maintain the orthogonality of the weight matrix would help prevent exploding and vanishing gradient, such as in RNN. But this regularization seems to fail that. So my question is, why would anyone want to use soft orthogonal regularization? All it seems to be doing is to keep $W$ to stray too far from being orthogonal. Is there any benefit in doing this?

  • $\begingroup$ Can you please cite the papers you are reading? Moreover, can you explain these conclusions 1. "This suggests that after the first update, there is no penalty for orthogonality..." and 2. "But this regularization seems to fail that"? $\endgroup$ – nbro Nov 22 '20 at 11:34
  • $\begingroup$ Is this Can We Gain More from Orthogonality Regularizations in Training Deep CNNs? the paper you are reading? Are you reading other papers? Please, edit your post to include these details. $\endgroup$ – nbro Nov 23 '20 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.