Why are thresholds in NN so important?

I am working on a paper on the approximation properties of NN. One part of this topic is of course the universal approximation theorem (UAT), which gets discussed in various papers with different properties for the activation function.

In the paper "Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function" by Leshno, they mention the importance of thresholds and follow that you don't need a continuous or smooth activation function for a universal approximator, because other papers already have proven the density of $$span\{f(a \cdot x) : \mathbb{R}^d \to \mathbb{R} \ | \ a \in \mathbb{R}^d, f \in C(\mathbb{R})\}$$ in $$C(\mathbb{R^d})$$ and in there version of the UAT the activation function $$\varrho$$ just needs to be not a polynomial (also essentially bounded and almost everywhere continuous, but they seem to ignore this in their reasoning).

I understand that for the UAT I just need one function for density, instead of all continuous functions as in the density result described, and the activation function is also just almost everywhere continuous, but has maybe someone some ideas or inputs why the threshold is so important?

Note: My definition of a 2-layered NN, with the architecture $$((d,n, 1 ), \varrho)$$, meaning we have $$d$$ inputs, $$n$$ hidden units and $$1$$ output, is $$\mathcal{F}_{((d,n, 1 ), \varrho)}=\{\sum_{i=1}^n a_i\varrho(w_i\cdot x+b_i) + c_i \ | \ b_i, c_i \in \mathbb{R} \text{ and } w_i \in \mathbb{R}^d\}$$. In this instance the $$b_i$$ are the thresholds.

• what are you referring to when you say “threshold”? Sep 29, 2023 at 16:20
• It is also called bias. I am going to include a definition of them in the post. Sep 30, 2023 at 11:16
• It's hard to know how to answer this without elaboration on the sense in which the thresholds are "important." Important for what? But perhaps you have overlooked that you can absorb the $b_i$ in $w_i$ if all $x$ contain a constant $1$. Oct 1, 2023 at 0:31
• Yeah, but it still acts like a threshold. I have an example to explain what I mean a bit better: Consider the activation function $f(x)=sin(x)$, then the hypothesis set, not considering thresholds, $\{sin(ax) : a \in \mathbb{R}\}$, only consists of odd functions. (For simplicity reasons we only consider networks with 1 input, 1 neuron in hidden layer, 1 output) Then there is no way to approximate an even function, like the Cosine. Only if we add a bias, in this case $b=\pi/2$. Oct 2, 2023 at 9:34
• Just to clarify I asked this question, because maybe someone has some knowledge about thresholds or examples, so I gain some more insight. Oct 2, 2023 at 9:36