1
$\begingroup$

Lately, I have been reading a lot about the universal approximation theorem. I was surprised to find only theorems about "single-channel" standard networks (multi-layer perceptrons), where all layers are 2D vectors and the weights can be represented in weight matrices.

In particular, this is no longer applicable in some convolutional network applications, where the layers tend to be tensors with multiple feature channels. Of course, one could construct an equivalent "single-channel" neural network from a multi-channel network by putting the weight matrices together in a certain way. However, one would then have sparse matrices as weight matrices with very many constraints on the matrix entries, so that the "standard theorems" would no longer be applicable.

Do you know of any papers that study the Universal Approximation Theorem for multi-channel neural networks? Or is there a way to derive it from one of the other theorems?

$\endgroup$
1
  • $\begingroup$ For "multi-channel neural networks" do you mean (just) convolutional neural networks? If not, can you please clarify what other "multi-channel" neural networks are you referring to? $\endgroup$
    – nbro
    Nov 2 '21 at 22:51
1
$\begingroup$

Yes, there is such a statement, valid even in a bit more general setting.

Any function, equivariant to a certain symmetry, can be approximated arbitrarily well, provided that the number of parameters is sufficient.

For the case of CNN, the symmetry is translational, and with a sufficient number of filters, any translationally equivariant function can be approximated by a suitable CNN.

For the details and exposition, please, look at the paper Universal approximations of invariant maps by neural networks (2018) by Dmitry Yarotsky.

$\endgroup$
2
  • $\begingroup$ Hello. I just want to let you know that, whenever you're providing a reference, I think it is a good idea to provide not just the link but also its name/title, because the link can become broken. $\endgroup$
    – nbro
    Nov 2 '21 at 12:35
  • $\begingroup$ @nbro sorry for that, I'll keep in mind) $\endgroup$ Nov 3 '21 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.