I'm trying to approximate the following function with a neural network (in Python).

# Train a neural network so that y = my_function(x0, x1, ...) ~ NN(x0, x1, ...)

def my_function(x0, x1, x2, x3, z1, z2, h1, h2):
    # Inputs are 1D arrays of type float64
    # And h1 and h2 are always >= 0.

    # b is always >= 0
    b = np.maximum(z1 + h1, z2 + h2) - np.maximum(z1, z2) 

    c = (z1 + h1) - (z2 + h2)
    term1 = .9 * x0 + 0.05 * (x1 + x2)
    term2 = 4.9 * b * c

    # if term1 * term2 < 0 then term1 = x0
    mask = term1 * term2 < 0 
    term1[mask] = x0[mask] 
    x_norm = np.sqrt(x0**2 + x3**2)
    term3 = 1. + 0.0044 * x_norm / np.power(b, 7./3.)
    return (term1 + term2) / term3

I generated 3 million rows of data to train a neural network. However, I do not seem to achieve a good approximation.

I tried many neural network structures using a combination of the following:

 Transfer uniformly Radom samples of input datausing  (x-mu)/sigma
 n_layers:  ranged between 3 to 5 layers
 n_neurons: 30 to 60
 Learning_rate: from 0.001 to 0.000001
 Activation_function = tried relu and and then tanh 
 batch_size = 15,000 to 40,000 
 loss_fnction = mean squared error

The lowest loss I get after around 9K epochs was 1.5464e-06. The following figure shows the residuals (y_pred - y_obs):

enter image description here

Am I missing something important here? What else I need to try to see if there will be an improvement?

  • 1
    $\begingroup$ Your function is not a trivial one. First, I would suggest that you write the mathematical equation with MathJax (or latex) rather than providing the implementation in Python because this site is not really appropriate for debugging code. See ai.stackexchange.com/help/on-topic for more details. In order to understand if your problem is with your neural network, loss, etc., I suggest you train the neural network to predict the output of a simpler function (e.g. the sine or a linear function). Then, in the next phase, train the neural network to learn a slightly more complicated function $\endgroup$ – nbro Jun 2 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.