# Why does the activation function for a hidden layer in a MLP have to be non-polynomial?

Across multiple pieces of literature describing MLPs or while describing the universal approximation theorem, the statement is very specific on the activation function being non-polynomial.

Is there a reason why it cannot be a higher-order polynomial? Is it just an attempt to use the least complex solution or we really cannot use higher-order polynomial?

I can understand the reason for the non-linear, but I am clueless about the non-polynomial requirement.

The paper Multilayer feedforward networks with a nonpolynomial activation function can approximate any function (by Leshno et al., 1993) provides a theorem claiming this and the (quite long) proof of the theorem in the appendix.

After a quick reading, it seems to me that they do not provide a very intuitive explanation of why the non-polynomiality of the (bounded, non-constant, and not necessarily continuous) activation function is necessary and sufficient (it's an if and only if result, see the last paragraph on p. 4) for a single-layer neural network to approximate any continuous function.

The theorem (p. 10) formally states that the set of all possible functions that the neural network can compute, denoted by $$\Sigma_n$$, is dense in $$C(\mathbb{R}^n)$$, the set of continuous functions from $$\mathbb{R}^n$$ to $$\mathbb{R}$$ (and being dense in $$C(\mathbb{R}^n)$$ is the equivalent mathematical statement to "neural networks can approximate any continuous function"). To understand this, you need to understand what a dense (sub)set is. To understand that, you need to understand what a closure of a set $$S$$ is: intuitively, it's the set of all points $$S$$ plus the points near the set $$S$$.

To prove this theorem (p. 12), they proceed in 7 steps, so it's a long proof.

For example, in step 1, they show (or just state) that, if the activation function $$\sigma$$ is a polynomial, then $$\Sigma_n$$ is not dense in $$C(\mathbb{R}^n)$$. They conclude that $$\Sigma_n$$, with such an activation function, would be a set of polynomials, which is not dense in $$C(\mathbb{R}^n)$$ (not sure why this is the case, but there is an old question here that asks exactly this; I think that understanding this would be a great step for intuitively understanding the theorem).

I don't plan to go over all the steps now, but, if you spend some time reading this paper, you should have the answer of why the non-polynomiality of the activation function is necessary and sufficient for the neural network to approximate any continuous function of the form $$f : \mathbb{R}^n \rightarrow \mathbb{R}$$. If that is not very useful, you could try reading the other related (but even longer) paper Approximation theory of the MLP model in neural networks (by Pinkus, 1999).

So, I wouldn't say that non-polynomiality is a complexity requirement, but really a theoretical requirement for approximating continuous functions.

(By the way, I think there's a typo on page 6. They write $$f_\omega : \mathbb{R} \rightarrow \mathbb{R}^n$$ to denote the function the neural network computes, but I am pretty sure they meant $$f_\omega : \mathbb{R}^n \rightarrow \mathbb{R}$$; in fact, previously, they assume that the neural network has $$n$$ inputs and $$1$$ output).

Like the accepted answer, I'm assuming you are referring to (literature refering to) Leshno et al, 1993. That paper only concerns 1-layer neural networks, and those are simply of the form $$x\mapsto\sum_{j=1}^Na_j\sigma(w_j\cdot x+\theta_j)\qquad(N\in\mathbb N,a_j\in\mathbb R,w_j\in\mathbb R^n,\theta_j\in\mathbb R).$$ I think of them as sums of scaled, dilated and rotated versions of the activation function $$\sigma$$. If the activation function is a polynomial of degree $$k$$ then every 1-layer neural network will be a polynomial of degree $$k$$ (and no higher). It is easy to construct a function that cannot be approximated (uniformly on the unit cube) by polynomials of degree $$k$$. Generically, polynomials of degree $$k+1$$ cannot be approximated by polynomials of degree $$k$$.

For example, take $$n=1$$ (one input dimension). Then $$f(x)= x^3$$ cannot be approximated by 2nd degree polynomials: the latter are all parabolas. So $$f$$ cannot be approximated (uniformly on $$[-1,1]$$) by 1-layer neural networks with activation function $$\sigma(x)=x^2$$.