Is it possible to control asymptotic behaviour of neural network models?

Question

Is it possible to specify what the asymptotic behaviour of a Neural Networks (NN) model should be?

I am thinking of a NN which tries to learn a mapping $\vec y=f(\vec x)$ with $\vec x$ a vector of features of dimension $d$ and $\vec y$ a vector of outputs of dimension $p$.

Is it possible to specify that, for instance, the NN should have a fixed value when $x_1$ goes to infinite?

I mean: $$ \lim_{x_1\to \infty} f(\vec x) = \vec c $$

If it is not possible with NN, do you know other machine learning models (for instance Gaussian Process Regression or Support Vector Regression) which have a known asymptotic behaviour?

Answer 1

A simpler answer is that for a standard neural net, the asymptotic behaviour is the asymptotic behaviour of the output neurons. For example, if the output layer is ReLUs, then the asymptotic behaviour is necessarily linear.

In your case, since you want it to be asymptotically constant, you can use the slightly old-fashioned choice of sigmoid units in the output layer.

The training set is necessarily finite, so no machine learning method can learn asymptotic behaviour. It can only be supplied by prior knowledge, for example as I describe.

Answer 2

For standard NNs, their extrapolation behavior an important aspect for financial applications cannot be controlled due to complex functional forms typically involved.

Neural Networks with Asymptotics Control discuss how they overcome this significant limitation and develop a new type of neural networks that incorporate large-value asymptotics, when known, allowing explicit control over extrapolation. This new type of asymptotics-controlled Neural network is based on two novel technical constructs:

1. A Multi-Dimensional Spline Interpolator with Prescribed Asymptotic Behavior
2. and A Custom NN layer that guarantees zero asymptotics in chosen directions.

Let $f(x)$ be the Function that we want to approximate while preserving its asymptotics. (It goes without saying that the asymptotics needs to be known.) As the first step, we find a control variate function $S(x)$ that has the same asymptotics as $f(x)$.

• Multi-Dimensional Inputs - the most interesting case - these asymptotics could either be known in all directions or only in some; naturally, as much information about the asymptotics as is available should be incorporated into the control variate function $S(x)$.

  • For step one, they show how to construct a universal control variate, a multi-dimensional spline $S(x)$ that has the same asymptotics as $f(x)$; and that can be used in all situations. In specific applications, of course, if more is known about the function $f$; a better choice of the control variate could be available.

  • For step two, we design a custom NN layer that guarantees zero asymptotics in all directions, with fine control over the regions where the NN interpolation is used and where the asymptotics kick in. they approximate the residual function $R(x) = S(x) - f(x)$ with a special NN that has vanishing (zero) asymptotics in all, or some, directions.

One-Dimensional Spline with Asymptotics Control:

The paper discussed that Instead of using the natural boundary conditions, i.e., setting second derivatives to zero at the boundaries, they have fixed the first derivatives at boundary points to arbitrary values. $\color{blue}{\text{Clearly, this paves the way to control asymptotics}}$.

Suppose that we know the behavior of the original, calculation-heavy function in its tails, i.e. $f(x) \simeq f_{−}(x)$ for $x<h_0$ and $f(x) \simeq f_{+}(x)$ for $x>h_{N+1}$ for large negative $h_0$ and large positive $h_{N+1}$. First, we expand the set of spline nodes to include points $h_0$ and $h_{N+1}$ and make sure the spline passes through them, $S(h_0) = f_{−}(h_0)$ and $S(h_{N+1}) = f_{+}(h_{N+1})$. To finish, they have specifies first order derivatives at these new boundary points.

$$S^{\prime}_0(h_0) = f^{\prime}_{−}(h_0), S(h_{N+1}) = f^{\prime}_{+}(h_{N+1})$$

This fully specifies the approximating function $\tilde{S}(x)$, $$\tilde{S}(x)=\left\{\begin{array}{ll}f_{-}(x), & x \leq h_{0} \\ S(x), & h_{0}<x<h_{N+1} \\ f_{+}(x), & h_{N+1} \leq x\end{array}\right.$$

One-Dimensional Spline with Asymptotics Control: Continuous Second Derivatives

where they have discussed how they have constructed a control variate function based on the spline like above, but with the continuous second derivative at the endpoints. In order to do that, they have picked a point between $h_0$ and $h_1$, say $h_{\frac{1}{2}}$ , and find, analytically, the value $S(h_{\frac{1}{2}})$ such that the second derivative at $h_{0}, S^{\prime \prime}\left(h_{0}\right)= f_{-}^{\prime \prime}\left(h_{0}\right)$ is matched.

Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems. This paper deals with studying the asymptotical properties of multilayer neural networks models used for the adaptive identification of the wide class of nonlinearly parameterized systems in a stochastic environment.

A study of the asymptotic behavior of neural networks. where they have studied neural networks modeled as a set of nonlinear differential equations of the form $TX+X=Wf(X)+b$, where $X$ is the neural membrane potential vector, $W$ is the network connectivity matrix, and $F(X)$ is the nonlinearity (an essentially sigmoid function). Topologies of neural networks that exhibit asymptotic behavior are established as behavior depends solely on the topology of the network. Moreover, the connectivity $W$ need not be symmetric. The simulated behavior of typical neural networks is presented

Refrence

Neural Networks with Asymptotics Control

Asymptotic Behaviour of Gradient Learning Algorithms in Neural Network Models for the Identification of Nonlinear Systems

A study of the asymptotic behavior of neural networks