4
$\begingroup$

Across multiple pieces of literature describing MLPs or while describing the universal approximation theorem, the statement is very specific on the activation function being non-polynomial.

Is there a reason why it cannot be a higher-order polynomial? Is it just an attempt to use the least complex solution or we really cannot use higher-order polynomial?

I can understand the reason for the non-linear, but I am clueless about the non-polynomial requirement.

$\endgroup$

1 Answer 1

3
$\begingroup$

The paper Multilayer feedforward networks with a nonpolynomial activation function can approximate any function (by Leshno et al., 1993) provides a theorem claiming this and the (quite long) proof of the theorem in the appendix.

After a quick reading, it seems to me that they do not provide a very intuitive explanation of why the non-polynomiality of the (bounded, non-constant, and not necessarily continuous) activation function is necessary and sufficient (it's an if and only if result, see the last paragraph on p. 4) for a single-layer neural network to approximate any continuous function.

The theorem (p. 10) formally states that the set of all possible functions that the neural network can compute, denoted by $\Sigma_n$, is dense in $C(\mathbb{R}^n)$, the set of continuous functions from $\mathbb{R}^n$ to $\mathbb{R}$ (and being dense in $C(\mathbb{R}^n)$ is the equivalent mathematical statement to "neural networks can approximate any continuous function"). To understand this, you need to understand what a dense (sub)set is. To understand that, you need to understand what a closure of a set $S$ is: intuitively, it's the set of all points $S$ plus the points near the set $S$.

To prove this theorem (p. 12), they proceed in 7 steps, so it's a long proof.

For example, in step 1, they show (or just state) that, if the activation function $\sigma$ is a polynomial, then $\Sigma_n$ is not dense in $C(\mathbb{R}^n)$. They conclude that $\Sigma_n$, with such an activation function, would be a set of polynomials, which is not dense in $C(\mathbb{R}^n)$ (not sure why this is the case, but there is an old question here that asks exactly this; I think that understanding this would be a great step for intuitively understanding the theorem).

I don't plan to go over all the steps now, but, if you spend some time reading this paper, you should have the answer of why the non-polynomiality of the activation function is necessary and sufficient for the neural network to approximate any continuous function of the form $f : \mathbb{R}^n \rightarrow \mathbb{R}$. If that is not very useful, you could try reading the other related (but even longer) paper Approximation theory of the MLP model in neural networks (by Pinkus, 1999).

So, I wouldn't say that non-polynomiality is a complexity requirement, but really a theoretical requirement for approximating continuous functions.

(By the way, I think there's a typo on page 6. They write $f_\omega : \mathbb{R} \rightarrow \mathbb{R}^n$ to denote the function the neural network computes, but I am pretty sure they meant $f_\omega : \mathbb{R}^n \rightarrow \mathbb{R}$; in fact, previously, they assume that the neural network has $n$ inputs and $1$ output).

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .