Since the Universal approximation theorem shows that standard multilayer feedforward networks with as few as a single hidden layer, sufficient hidden units, and arbitrary bounded and nonconstant activation function can approximate any continuous function with arbitrary accuracy, does this theory also apply to binary neural networks? Is there a specific mathematical proof that binary neural networks can fit any function.
$\begingroup$
$\endgroup$
2
-
3$\begingroup$ What do you mean by binary here. Do you mean a single output neuron with a sigmoid activation? Do you mean a two neuron output with some undefined activation? $\endgroup$– David HoelzerCommented Feb 12, 2023 at 13:01
-
$\begingroup$ @DavidHoelzer Binary neural network is a neural network after the extreme quantification of the general neural network, and the weight and activation in the network are only represented by 1bit. $\endgroup$– user68072Commented Feb 15, 2023 at 8:25
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Yes, for a broad class, they actually do, with probability 1:
[i] Wang, Yanzhi, et al. "Universal approximation property and equivalence of stochastic computing-based neural networks and binary neural networks." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019 https://arxiv.org/abs/1803.05391
[ii] Yayla, Mikail, et al. "Universal Approximation Theorems of Fully Connected Binarized Neural Networks." arXiv preprint arXiv:2102.02631 (2021).
