Lately, I have been reading a lot about the universal approximation theorem. I was surprised to find only theorems about "single-channel" standard networks (multi-layer perceptrons), where all layers are 2D vectors and the weights can be represented in weight matrices.
In particular, this is no longer applicable in some convolutional network applications, where the layers tend to be tensors with multiple feature channels. Of course, one could construct an equivalent "single-channel" neural network from a multi-channel network by putting the weight matrices together in a certain way. However, one would then have sparse matrices as weight matrices with very many constraints on the matrix entries, so that the "standard theorems" would no longer be applicable.
Do you know of any papers that study the Universal Approximation Theorem for multi-channel neural networks? Or is there a way to derive it from one of the other theorems?