I'm trying to approximate the following function with a neural network (in Python).
# Train a neural network so that y = my_function(x0, x1, ...) ~ NN(x0, x1, ...) def my_function(x0, x1, x2, x3, z1, z2, h1, h2): # Inputs are 1D arrays of type float64 # And h1 and h2 are always >= 0. # b is always >= 0 b = np.maximum(z1 + h1, z2 + h2) - np.maximum(z1, z2) c = (z1 + h1) - (z2 + h2) term1 = .9 * x0 + 0.05 * (x1 + x2) term2 = 4.9 * b * c # if term1 * term2 < 0 then term1 = x0 mask = term1 * term2 < 0 term1[mask] = x0[mask] x_norm = np.sqrt(x0**2 + x3**2) term3 = 1. + 0.0044 * x_norm / np.power(b, 7./3.) return (term1 + term2) / term3
I generated 3 million rows of data to train a neural network. However, I do not seem to achieve a good approximation.
I tried many neural network structures using a combination of the following:
Transfer uniformly Radom samples of input datausing (x-mu)/sigma n_layers: ranged between 3 to 5 layers n_neurons: 30 to 60 Learning_rate: from 0.001 to 0.000001 Activation_function = tried relu and and then tanh batch_size = 15,000 to 40,000 loss_fnction = mean squared error
The lowest loss I get after around 9K epochs was 1.5464e-06. The following figure shows the residuals (
y_pred - y_obs):
Am I missing something important here? What else I need to try to see if there will be an improvement?