We have convolutional neural networks and recurrent neural networks for analysing respectively images and sequential data.

How do I determine which neural network architecture is more appropriate to approximate a certain function? For example, suppose I want to approximate the function $f(x,y) = \sin(2\pi x)\sin(2\pi y)$ with domain $\Omega = [0,1]\times [0,1]$, that is, $x$ and $y$ can be between $0$ and $1$ (inclusive). For example, which kind of activation functions would be better suited for approximating this specific function?

  • 1
    $\begingroup$ All neural networks perform function approximation. Could you define what you mean by it in more detail? I see you have an actual function $f(x,y)$ here, but in practice you would never use a neural network when you both know the function and it is fast/easy to calculate as your example is (so the answer would be "the main architecture choice is to not use a neural network at all"). Do you have a specific non-trivial use in mind for your neural network? $\endgroup$ – Neil Slater Dec 3 '18 at 17:00

For the function you mentioned,

  • There has to be two input and one output neuron representing x and y values.

  • Use the ReLU function at the input and hidden layers.

  • Use linear activation function on the output layer.

  • This will create a regression architecture which will approximate the function as required.

| improve this answer | |
  • 4
    $\begingroup$ Can you justify your answer? To your first point, there should be TWO input neurons and ONE output neuron. $\endgroup$ – timudk Dec 3 '18 at 16:43
  • $\begingroup$ no relu here lol $\endgroup$ – user8426627 Jun 12 '19 at 12:48

If the concept class specified is

$$f(x, y) = k \, \sin(2 \pi f_x x) \, sin(2 \pi f_y y) \\ \land 0 < x < 1 \\ \land 0 < y < 1 \; \text{,}$$

and the optimum fit to example data is expected occur when $k \approx 1 \land f_x \approx 1 \land f_y \approx 1$, then it is not an AI problem. It is a problem that can be solved with a least squares convergence, probably in conjunction with a Fourier transform.

If nothing is known about $f(x, y)$ except continuity and that it is single valued with respect to $(x, y)$, then few conclusions can be drawn about best approach. In such a case, the domain of $x$ and $y$ are irrelevant because they can be normalized. Furthermore, the tree of operations, such as $\sin()$ and multiplication, are irrelevant too, because the function could just as easily be

$$f(x, y) = \ln(x) + \Gamma(y) - k \, \text{.}$$

The question indicates the design involves CNN and RNN components for analyzing images and sequential data. It is not clear whether the CNN is for the discovery of objects or waves (given the $\sin()$ in the function mentioned) and whether those objects move between frames so that the RNN must detect motion.

Nothing is given about the pool of example data available or planned to be available or the expected outputs of the system. If data is sequential, where is $t$ in the function? What is the objective of image analysis?

Although a deep MLP (multilayer perceptron) with SGD can learn an arbitrary function, it is hardly an architecture, the mention of images, CNN, RNN, and sequential data, MLP with SGD does not seem to match.

Regarding activation functions, the inner layer functions would depend on the higher level design requirements. The activation functions of the last layer of a single artificial network is usually chosen to match the data type and range of desired output for each output channel (dimension).

If the objective of this question is to take images and sequential data and produce something useful without a priori defining what useful means, then it is an unsolved AI problem thus far and no known topology comprised of artificial networks and other AI building blocks provide a solution. The autonomous development of internal concepts of usefulness would need to be developed mathematically and algorithmically and become practically speed optimized in hardware and software first.

| improve this answer | |

For example, suppose I want to approximate the function mashine learning is used to approximate an UNKNOWN function. If you do want to do this with NN, you will have a regression task with small full connected network with like:

  • Input is 2 values of range 0 ..1
  • a few hidden layers, just start trying with one, of size like 3-6 neurons , activation sigmoid or tanh
  • need a non-linear function here to approx non linear sin * sin, whatever
  • last layer with no activation function just making a difference from... known value f(x,y)
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.