Why can't neural networks learn functions outside of the specified domains?

Question

I understand that neural nets are fundamentally interpolative tools. Meaning, given a training dataset, a well trained neural net can approximate values within the domain of the training dataset. However, we are unsure about their behavior once we test against values outside that domain.

Speaking in the context of Imagenet, a NN trained on one of the classes in Imagenet will probably be able to predict an image of the same class outside Imagenet because Imagnet itself covers a huge domain for each class that whatever image we come across in the wild, its features will be accounted for by Imagnet.

Now, this intuition breaks down for me when I talk about simple functions with simple inputs. For example, consider $sin(x)$. Our goal is to train a neural net to predict the function given $x$ with a training domain $[-1, 1]$. Theoretically, the neural net should not be able to predict the values well outside that domain, right? This seems counterintuitive to me because the function behaves in a very simple and periodic way that I find it hard to believe that a neural net cannot figure out the proper transformation of that function even outside the training domain.

In short, are neural nets inherently unable to find a generalizable transformation outside the training domain no matter how simple is the function we are trying to approximate? Is this a property of the Deep Learning framework?

Are there problems where researchers were able to learn a robust generalizable transformation using neural nets outside the training domain? What are the possible conditions so that such results can happen?

Answer 1

The problem you discuss extends past the machine but to the man behind the machine (or woman). ML can be broken down into 3 components, the model, the data, and the learning procedure. This by the way extends to us as well. The model is our brain, the data is our experience and sensory input, and the learning procedure is there but unknown (for now $<$insert evil laugh$>$).

Your model's inability to understand that it is a sin function is normally by construction. First, though lets start with the fact that if someone showed me $sin(x)$ on $[-1,1]$, a sinusoid is the last thing id think of:

It looks almost linear. So for sake of argument i'm going to continue on the assumption you meant an entire period ($[-\pi, \pi])$.

Now, given this most people who ever got past pre-calculus would assume sinusoid, as youd expect, but let me show you how unstable that idea is. Lets use a full period, shifted by $\frac{\pi}{2}$
Now i don't know about you but my spidey-sense would make me think of a Gaussian before anything. This is the same sinusoid along an entire period but just the slight difference changes an entire perspective.

Now lets talk about some of the mathematics. Given the uncountable $(x,sin(x))$ pairs between $[-1, 1]$, we can extract the function $sin(x)$ with certainty (assuming infintely differentiable) by simply taking the taylor series along any of the points in that domain. More so, we only need an infinitiscemally small continuous subset of the function to achieve that result, but as a person, would you be able to tell from $[-.0001, .0001]$? (i don't think so!), because by default we quantize, and what were gonna give to the computer is a quantization as well. We are giving it a finite countable set of points and expect it to to generalize to an uncountable set. This is silly, since there exists an infinite set of functions that can fit those exact points (concept of overfitting steps in here).

So if its so unfeasible for a computer to correctly solve the function, why can us humans extrapolate it so well from the $[-\pi, \pi]$ domain? The answer is our preset biases. Growing up we have dealt so much with periodicity, specifically sins and cosines, that graphed period triggered something in our brains and we just know what it is. But this isnt always a benefit, i could draw you a low-ordered polynomial on that domain that would make you think that exact same thought making you wrong.

So going back to my initial point, the problems lies with the models creator and not the model. Like i explained just now, its unfeasible without preset biases to learn the function we want, so what should we do? give up? NO, its our goal as the model architect to figure out what we want and how we can best achieve it! so if we want it to accomplish this task, lets try our best in modeling these biases. A quick example of how you could do that, is to force the function to fit the fourier coefficients, thatll probably solve this quite quickly (ofcourse this limits this models ability to solve for other common functions)

Takeaway: It's not the model thats struggling, its us using preset biases that the model does not have access to. The solution to this is be clever and figure out how to have it either start with these preset biases or learn them.