Which neural network should I use to approximate a specific but unknown function?

Question

We have convolutional neural networks and recurrent neural networks for analyzing, respectively, images and sequential data.

Now, suppose I want to approximate the unknown function $f(x,y) = \sin(2\pi x)\sin(2\pi y)$, with domain $\Omega = [0,1]\times [0,1]$, that is, $x$ and $y$ can be between $0$ and $1$ (inclusive).

How do I determine which neural network architecture is more appropriate to approximate this function? Which kind of activation functions would be better suited for this?

Note that, generally, I don't know a priori which function the neural network has to learn. I am just asking for this specific $f(x, y)$, as it could be a solution for a differential equation. And $\Omega$ is the domain, i.e., I don't care about the output of the neural network outside $\Omega$.

Answer 1

If the concept class specified is

$$f(x, y) = k \, \sin(2 \pi f_x x) \, sin(2 \pi f_y y) \\ \land 0 < x < 1 \\ \land 0 < y < 1 \; \text{,}$$

and the optimum fit to example data is expected occur when $k \approx 1 \land f_x \approx 1 \land f_y \approx 1$, then it is not an AI problem. It is a problem that can be solved with a least squares convergence, probably in conjunction with a Fourier transform.

If nothing is known about $f(x, y)$ except continuity and that it is single valued with respect to $(x, y)$, then few conclusions can be drawn about best approach. In such a case, the domain of $x$ and $y$ are irrelevant because they can be normalized. Furthermore, the tree of operations, such as $\sin()$ and multiplication, are irrelevant too, because the function could just as easily be

$$f(x, y) = \ln(x) + \Gamma(y) - k \, \text{.}$$

The question indicates the design involves CNN and RNN components for analyzing images and sequential data. It is not clear whether the CNN is for the discovery of objects or waves (given the $\sin()$ in the function mentioned) and whether those objects move between frames so that the RNN must detect motion.

Nothing is given about the pool of example data available or planned to be available or the expected outputs of the system. If data is sequential, where is $t$ in the function? What is the objective of image analysis?

Although a deep MLP (multilayer perceptron) with SGD can learn an arbitrary function, it is hardly an architecture, the mention of images, CNN, RNN, and sequential data, MLP with SGD does not seem to match.

Regarding activation functions, the inner layer functions would depend on the higher level design requirements. The activation functions of the last layer of a single artificial network is usually chosen to match the data type and range of desired output for each output channel (dimension).

If the objective of this question is to take images and sequential data and produce something useful without a priori defining what useful means, then it is an unsolved AI problem thus far and no known topology comprised of artificial networks and other AI building blocks provide a solution. The autonomous development of internal concepts of usefulness would need to be developed mathematically and algorithmically and become practically speed optimized in hardware and software first.

Answer 2

For example, suppose I want to approximate the function mashine learning is used to approximate an UNKNOWN function. If you do want to do this with NN, you will have a regression task with small full connected network with like:

• Input is 2 values of range 0 ..1
• a few hidden layers, just start trying with one, of size like 3-6 neurons , activation sigmoid or tanh
• need a non-linear function here to approx non linear sin * sin, whatever
• last layer with no activation function just making a difference from... known value f(x,y)