I'm reading papers regarding soft orthogonal regularization, $\frac \lambda 4||WW^\intercal - I||_F^2$, over a deep neural network whose activation function is ReLU and weight matrix $W$ is initialized orthogonal.
To my understanding, the orthogonal penalty's purpose is to encourage $W$ to become more orthogonalized with each update. But the derived total gradient would be $\nabla = \lambda W(WW^\intercal - I) + \nabla H$, for $H$ be the loss function. This suggests that after the first update, there is no penalty for orthogonality, and thus it is unlikely that $W$ will maintain its orthogonality.
The ability to maintain the orthogonality of the weight matrix would help prevent exploding and vanishing gradient, such as in RNN. But this regularization seems to fail that. So my question is, why would anyone want to use soft orthogonal regularization? All it seems to be doing is to keep $W$ to stray too far from being orthogonal. Is there any benefit in doing this?
