Since the Universal approximation theorem shows that standard multilayer feedforward networks with as few as a single hidden layer, sufficient hidden units, and arbitrary bounded and nonconstant activation function can approximate any continuous function with arbitrary accuracy, does this theory also apply to binary neural networks? Is there a specific mathematical proof that binary neural networks can fit any function.
Yes, for a broad class, they actually do, with probability 1:
[i] Wang, Yanzhi, et al. "Universal approximation property and equivalence of stochastic computing-based neural networks and binary neural networks." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019 https://arxiv.org/abs/1803.05391
[ii] Yayla, Mikail, et al. "Universal Approximation Theorems of Fully Connected Binarized Neural Networks." arXiv preprint arXiv:2102.02631 (2021).