Can someone give me the main idea of the paper Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function? I'm having trouble understanding it.
Polynomials are unbounded once the input variable is very large or negative, also most feedforward NNs are using backpropagation algorithms to adjust weights during each training iteration which needs the derivatives of all the activation functions and thus possibly leads to unbounded gradient if any of these are mere polynomials.
And from the abstract of your referenced paper threshold plays an important role in universal function approximator, an obvious easy example is the 2 layer perceptrons (aka threshold logic unit [TLU], Adaline [Widrow, 1962]) neural network implementation of the classic Boolean even-parity function $y=x_1x_2+\bar x_1 \bar x_2$ which is not linearly separable and thus a single layer of perceptrons cannot achieve as pointed out by Minsky in the 60's，but it could be easily solved by below 2 layer perceptrons with all (bounded) step functions assigned with appropriate thresholds as shown below (you can manually try to verify). If all activation functions are mere polynomials, no matter how many hidden layers you add, these Boolean functions cannot be modeled exactly, and, thus, most practical classification problems could not be solved by neural networks.