I'm reading the following paper in which the author seems to do 2 things interesting:
- The hidden-to-hidden weight matrix of the RNN is SVD decomposed and train separately.
- Each orthogonal part of the decomposition is optimized multiplicatively according to Cayley Transformation to maintain its orthogonal properties.
Now, I'm not so strong with the math behind the technique, but I could be hand-waving and say that albeit being multiplicative, it is just another method of gradient descent, and each orthogonal part is still minimizing the Loss function. So far so good.
But what they are doing is actually split the original optimization problem into multiple sub-optimization (2 for orthogonal matrices and n for the number of singular values), and then multiplied the result together. How can we be sure about the convergence and the optimality of such method? Or is this the case where we can say nothing and let the experiment speak for themselves?