I am working on a paper on the approximation properties of NN. One part of this topic is of course the universal approximation theorem (UAT), which gets discussed in various papers with different properties for the activation function.

In the paper "Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function" by Leshno, they mention the importance of thresholds and follow that you don't need a continuous or smooth activation function for a universal approximator, because other papers already have proven the density of $span\{f(a \cdot x) : \mathbb{R}^d \to \mathbb{R} \ | \ a \in \mathbb{R}^d, f \in C(\mathbb{R})\}$ in $C(\mathbb{R^d})$ and in there version of the UAT the activation function $\varrho$ just needs to be not a polynomial (also essentially bounded and almost everywhere continuous, but they seem to ignore this in their reasoning).

I understand that for the UAT I just need one function for density, instead of all continuous functions as in the density result described, and the activation function is also just almost everywhere continuous, but has maybe someone some ideas or inputs why the threshold is so important?

Note: My definition of a 2-layered NN, with the architecture $((d,n, 1 ), \varrho)$, meaning we have $d$ inputs, $n$ hidden units and $1$ output, is $\mathcal{F}_{((d,n, 1 ), \varrho)}=\{\sum_{i=1}^n a_i\varrho(w_i\cdot x+b_i) + c_i \ | \ b_i, c_i \in \mathbb{R} \text{ and } w_i \in \mathbb{R}^d\}$. In this instance the $b_i$ are the thresholds.

  • 1
    $\begingroup$ what are you referring to when you say “threshold”? $\endgroup$
    – Alberto
    Sep 29, 2023 at 16:20
  • $\begingroup$ It is also called bias. I am going to include a definition of them in the post. $\endgroup$
    – Lopsio
    Sep 30, 2023 at 11:16
  • $\begingroup$ It's hard to know how to answer this without elaboration on the sense in which the thresholds are "important." Important for what? But perhaps you have overlooked that you can absorb the $b_i$ in $w_i$ if all $x$ contain a constant $1$. $\endgroup$
    – Sycorax
    Oct 1, 2023 at 0:31
  • $\begingroup$ Yeah, but it still acts like a threshold. I have an example to explain what I mean a bit better: Consider the activation function $f(x)=sin(x)$, then the hypothesis set, not considering thresholds, $\{sin(ax) : a \in \mathbb{R}\}$, only consists of odd functions. (For simplicity reasons we only consider networks with 1 input, 1 neuron in hidden layer, 1 output) Then there is no way to approximate an even function, like the Cosine. Only if we add a bias, in this case $b=\pi/2$. $\endgroup$
    – Lopsio
    Oct 2, 2023 at 9:34
  • $\begingroup$ Just to clarify I asked this question, because maybe someone has some knowledge about thresholds or examples, so I gain some more insight. $\endgroup$
    – Lopsio
    Oct 2, 2023 at 9:36


You must log in to answer this question.

Browse other questions tagged .