Suppose we want to estimate a continuous function $f:\mathbb R^2 \rightarrow \mathbb R$ based on a sample using a NN (around 1000 examples). This function is not bounded. Which architecture would you choose ? How many layers/neurons ? Which activation functions ? Which loss function ?
Intuitively, I would go with one hidden layer, 2 neurons, $L^2$ loss, and maybe the Bent identity for the output and a sigmoid in the hidden layer ?
What are the advantages of doing something "fancier" than that ?
Would you also have chosen to use a NN for this job or would you have considered a regression SVM for example or something else (knowing that precision is the goal)?