2
$\begingroup$

Suppose we want to estimate a continuous function $f:\mathbb R^2 \rightarrow \mathbb R$ based on a sample using a NN (around 1000 examples). This function is not bounded. Which architecture would you choose ? How many layers/neurons ? Which activation functions ? Which loss function ?

Intuitively, I would go with one hidden layer, 2 neurons, $L^2$ loss, and maybe the Bent identity for the output and a sigmoid in the hidden layer ?

What are the advantages of doing something "fancier" than that ?

Would you also have chosen to use a NN for this job or would you have considered a regression SVM for example or something else (knowing that precision is the goal)?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It depends on the complexity of your problem. $\mathbb{R}^2 \rightarrow \mathbb{R}^1$ looks simple, but I can give you some nonsense complicated examples that need a deep network. So, the complexity of the problem sets the number of layers and neurons. The kind of problem will determine the architecture of your network (if it needs memory or not). In most cases, the mean squared error is ok. However, for the activation function, I would go with the ReLU.

If the SVM is good enough for your problem, then go with it. Your question is general and an exact answer needs more information about the problem.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Previously, you had written that the ReLU is better. I removed the "better", but maybe you should explain why you're suggesting the ReLU. Maybe you should also mention the universal approximation theorems, given the generality of the question, if you heard of them. $\endgroup$
    – nbro
    Commented Dec 15, 2020 at 21:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .