2
$\begingroup$

As stated in the universal approximation theorem, a neural network can approximate almost any function.

Is there a way to calculate the closed-form (or analytical) expression of the function that a neural network computes/approximates?

Or, alternatively, figure out if the function is linear or non-linear?

$\endgroup$
2
$\begingroup$

To check if a function is linear is easy: if you can train one fully connected layer, without activations, of the right dimensions (for a function $\mathbb{R}^n \rightarrow \mathbb{R}^m$ you need $nm$ weights aka the matrix corresponding to the linear application), with enough data, to 100% accuracy... then it is linear.

The estimated function is explicit: it is given by the architecture of the NN and its weights. I don't think you can hope for an answer like "$\sin(\pi x + \phi) +4$" if it's not already in the predictive capacity of the NN. Interpretability of NNs is a hot topic. In what cases symbolic reasoning can lead to a simplified expression?

$\endgroup$
1
  • 1
    $\begingroup$ You actually have the analytical expression of the function that the neural network computes, even if that neural network computes a non-linear function. Take a look at this example: i.e. it's just the expression that you use to perform the forward pass of the neural network. The problem is how to interpret this function, that's why neural networks are black-box models, because they are not easily interpretable. $\endgroup$ – nbro Jan 3 at 20:33
1
$\begingroup$

The network is the function.

A network is a function, that is modeled by terms describing the architecture and coefficients that are learned.

Look at a simple model:

$$f(x) = ax+b$$

Your solver determines $a$ and $b$, and you substitute them into $f(x)$ and then you're able to calculate $f(42)$. The function is linear by definition, but may not be a good fit for your data.

To see if the input data may belong to a linear function can be achieved, when you have a network, that can both fit linear functions and higher order functions. For example try to fit your data to

$$f(x)=ax^2+bx+c$$

The function is linear, when $a=0$. If the function is quadratic you get $a\ne 0$ and if it has a higher order than $2$ you will get not good fit with any $a, b, c$.

When looking at MLP and similar networks, the question "is it linear" may not be that easy to answer, but the question how to get the function is the same: Your trained network is the function.

I guess one option to see if the function approximated by an existing network is linear is to train a second network that only contains linear elements. If it is able to approximate your first network, the first network is linear as well. Of course you will need the right test data, as you may accidentally choose a training set that has linearly correlated output in a non-linear function. This is of course the same problem as always, that your network will always can only be as good as the training data.

$\endgroup$
0
$\begingroup$

As opposed to what is written in this answer, you can have the analytical expression of the function that the neural network computes, even if that neural network computes a non-linear function. Take a look at this example: i.e. it is just the expression that you use to perform the forward pass of the neural network. The problem is how to interpret this function, how it generally behaves, and how it is different from the usually unknown function that the neural network is supposed to approximately compute: that's why neural networks are called black-box models, because they are not easily interpretable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.