2
$\begingroup$

Can someone give me the main idea of the paper Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function? I'm having trouble understanding it.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Polynomials are unbounded once the input variable is very large or negative, also most feedforward NNs are using backpropagation algorithms to adjust weights during each training iteration which needs the derivatives of all the activation functions and thus possibly leads to unbounded gradient if any of these are mere polynomials.

And from the abstract of your referenced paper threshold plays an important role in universal function approximator, an obvious easy example is the 2 layer perceptrons (aka threshold logic unit [TLU], Adaline [Widrow, 1962]) neural network implementation of the classic Boolean even-parity function $y=x_1x_2+\bar x_1 \bar x_2$ which is not linearly separable and thus a single layer of perceptrons cannot achieve as pointed out by Minsky in the 60's，but it could be easily solved by below 2 layer perceptrons with all (bounded) step functions assigned with appropriate thresholds as shown below (you can manually try to verify). If all activation functions are mere polynomials, no matter how many hidden layers you add, these Boolean functions cannot be modeled exactly, and, thus, most practical classification problems could not be solved by neural networks.

enter image description here

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Do you think you can provide a simple example why a polynomial nonlinearity would lead to an unbounded gradient? I don't see how that could be the case. (The derivative of a polynomial is just a polynomial, not infinity) $\endgroup$
    – Steven Sagona
    Nov 22, 2022 at 16:17
  • $\begingroup$ @StevenSagona Here unbounded just means without a pre-assigned fixed threshold, (higher order) polynomial activation functions may contain bilinear terms which could complicate and amplify the (partial) derivatives. For example see this paper about tensor network: the mathematical explanation of neural tensor networks remains a challenging problem, due to the bilinear term... we associate NTN with Taylor’s theorem and find that each slice of NTN could be represented as a 2nd order multivariate Taylor polynomial. $\endgroup$
    – Double Knot
    Nov 22, 2022 at 18:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .