Is there any paper that shows that multi-channel neural networks are universal approximators?

Question

Lately, I have been reading a lot about the universal approximation theorem. I was surprised to find only theorems about "single-channel" standard networks (multi-layer perceptrons), where all layers are 2D vectors and the weights can be represented in weight matrices.

In particular, this is no longer applicable in some convolutional network applications, where the layers tend to be tensors with multiple feature channels. Of course, one could construct an equivalent "single-channel" neural network from a multi-channel network by putting the weight matrices together in a certain way. However, one would then have sparse matrices as weight matrices with very many constraints on the matrix entries, so that the "standard theorems" would no longer be applicable.

Do you know of any papers that study the Universal Approximation Theorem for multi-channel neural networks? Or is there a way to derive it from one of the other theorems?

Answer 1

Yes, there is such a statement, valid even in a bit more general setting.

Any function, equivariant to a certain symmetry, can be approximated arbitrarily well, provided that the number of parameters is sufficient.

For the case of CNN, the symmetry is translational, and with a sufficient number of filters, any translationally equivariant function can be approximated by a suitable CNN.

For the details and exposition, please, look at the paper Universal approximations of invariant maps by neural networks (2018) by Dmitry Yarotsky.