2
$\begingroup$

As stated in the Universal approximation theorem, a neural network can approximate almost any function.

I was wondering, if there are methods to return the actual formula of the function that the NN estimated?

Or, alternatively, figure out if the function is linear or non-linear?

$\endgroup$
2
$\begingroup$

To check if a function is linear is easy: if you can train one fully connected layer, without activations, of the right dimensions (for a function $\mathbb{R}^n \rightarrow \mathbb{R}^m$ you need $nm$ weights aka the matrix corresponding to the linear application), with enough data, to 100% accuracy... then it is linear.

The estimated function is explicit: it is given by the architecture of the NN and its weights. I don't think you can hope for an answer like "$\sin(\pi x + \phi) +4$" if it's not already in the predictive capacity of the NN. Interpretability of NNs is a hot topic. In what cases symbolic reasoning can lead to a simplified expression?

$\endgroup$
1
$\begingroup$

The network is the function.

A network is a function, that is modeled by terms describing the architecture and coefficients that are learned.

Look at a simple model:

$$f(x) = ax+b$$

Your solver determines $a$ and $b$, and you substitute them into $f(x)$ and then you're able to calculate $f(42)$. The function is linear by definition, but may not be a good fit for your data.

To see if the input data may belong to a linear function can be achieved, when you have a network, that can both fit linear functions and higher order functions. For example try to fit your data to

$$f(x)=ax^2+bx+c$$

The function is linear, when $a=0$. If the function is quadratic you get $a\ne 0$ and if it has a higher order than $2$ you will get not good fit with any $a, b, c$.

When looking at MLP and similar networks, the question "is it linear" may not be that easy to answer, but the question how to get the function is the same: Your trained network is the function.

I guess one option to see if the function approximated by an existing network is linear is to train a second network that only contains linear elements. If it is able to approximate your first network, the first network is linear as well. Of course you will need the right test data, as you may accidentally choose a training set that has linearly correlated output in a non-linear function. This is of course the same problem as always, that your network will always can only be as good as the training data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.