# Why is my neural network not able to approximate this function?

I'm trying to approximate the following function with a neural network (in Python).

# Train a neural network so that y = my_function(x0, x1, ...) ~ NN(x0, x1, ...)

def my_function(x0, x1, x2, x3, z1, z2, h1, h2):
# Inputs are 1D arrays of type float64
# And h1 and h2 are always >= 0.

# b is always >= 0
b = np.maximum(z1 + h1, z2 + h2) - np.maximum(z1, z2)

c = (z1 + h1) - (z2 + h2)
term1 = .9 * x0 + 0.05 * (x1 + x2)
term2 = 4.9 * b * c

# if term1 * term2 < 0 then term1 = x0
mask = term1 * term2 < 0
x_norm = np.sqrt(x0**2 + x3**2)
term3 = 1. + 0.0044 * x_norm / np.power(b, 7./3.)
return (term1 + term2) / term3


I generated 3 million rows of data to train a neural network. However, I do not seem to achieve a good approximation.

I tried many neural network structures using a combination of the following:

 Transfer uniformly Radom samples of input datausing  (x-mu)/sigma
n_layers:  ranged between 3 to 5 layers
n_neurons: 30 to 60
Learning_rate: from 0.001 to 0.000001
Activation_function = tried relu and and then tanh
batch_size = 15,000 to 40,000
loss_fnction = mean squared error


The lowest loss I get after around 9K epochs was 1.5464e-06. The following figure shows the residuals (y_pred - y_obs):

Am I missing something important here? What else I need to try to see if there will be an improvement?

• Your function is not a trivial one. First, I would suggest that you write the mathematical equation with MathJax (or latex) rather than providing the implementation in Python because this site is not really appropriate for debugging code. See ai.stackexchange.com/help/on-topic for more details. In order to understand if your problem is with your neural network, loss, etc., I suggest you train the neural network to predict the output of a simpler function (e.g. the sine or a linear function). Then, in the next phase, train the neural network to learn a slightly more complicated function – nbro Jun 2 at 15:00