I'm reading this paper: "Orthogonalizing Convolutional Layers with the Cayley Transform" regarding the orthogonalization of convolutional layers, i.e. enforcing learning of orthogonal feature maps while training a CNN.
The math used to prove the effectiveness of the proposed method is not per se the problem. My doubt is about how to apply the proposed method to an entire network.
More specifically, looking at algorithm 1, page 5, we can see that the proposed method requires an input tensor to perform the orthogonalization. Moreover, the output of the algorithm is the "modified" input tensor, the weights of the convolutions are used but not updated explicitly. So my questions are:
- Am I supposed to apply the proposed procedure to every convolutional layer of my architecture during training?
- Will the network learn orthogonal feature maps as a result of backpropagation rather than thanks to the suggested method? (or to phrase it differently, does the method enforce orthogonal weights by modifying the input tensor thanks to backpropagation?)
Thanks in advance to whoever will provide some hints, I link down here some extra resources: