I have been trying to adjust a neural network to a simple function: the mass of an sphere. I have tried with different architectures, for example, a single hidden layer and two hidden layers, always with 128 neurons each, and training them for 5000 epochs. The code is the usual one. Just in case, I publish one of them

model = keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])
                        ,keras.layers.Dense(128, activation="relu")
                        ,keras.layers.Dense(1, activation="relu")])
model.compile(optimizer='rmsprop', loss='mse', metrics=['mae'])
history = model.fit(x, y, validation_split=0.2, epochs=5000)

The results are shown in the graphs.

two hidden layers, 128 neurons each

one hidden layer, 128 neurons

I suspect that I am making an error somewhere, because I have seen that deep learning is able to match complex functions with much less epochs. I shall appreciate any hint to fix this problem and obtain a good fit with the deep learning function.

In order to make it clear I post the graph's code.

rs =[x for x in range(20)]
def masas_circulo(x):
    masas_circulos =[]
    rs =[r for r in range(x)]
    for r in rs:

   return masas_circulos

 masas_circulos = masas_circulo(20) 
 esferas = [4/3*np.pi*r**3 for r in range(20)]
 import matplotlib.pyplot as plt
 plt.title("Mass of an sphere.\nDL (1hl,128 n,5000 e) vs ground_truth")
  • $\begingroup$ What are you trying to predict?: is it $V$ given $r$? for the formula $V=4/3\pi r^3$? if so, is it the output of your network, the single scalar, expected to be $V$? For me that seems like a plotting error: check how you are drawing the output of your network for different $r$ $\endgroup$
    – JVGD
    Jul 26 at 13:06
  • 1
    $\begingroup$ Ies, I am trying to predict V given r for the formula V=4/3πr^3. I have added the graphs' code. $\endgroup$ Jul 26 at 13:15
  • $\begingroup$ Sphere's density=1 😄 $\endgroup$ Jul 26 at 14:21

You're trying to learn a cubic function that explodes in values and your issue is scaling. I have been able to learn a better approximation by scaling data and using tanh as activation function.

Code and result are as below:

enter image description here

Convergence around X=100 happens because of tanh activation. Relu will not work better because of negative values that is the result of scaling. You can try playing with Leaky Relu activation and various alpha values.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

def mass_of_sphere(R):
    return (4/3) * np.pi * (R**3)

X = np.linspace(1, 120, 500000)
y = [mass_of_sphere(x) for x in X]

X = np.array(X).reshape(-1, 1)
y = np.array(y)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

scaler_X = StandardScaler()
X_train = scaler_X.fit_transform(X_train)
X_test = scaler_X.transform(X_test)

scaler_y = StandardScaler()
y_train = scaler_y.fit_transform(y_train.reshape(-1, 1))
y_test = scaler_y.transform(y_test.reshape(-1, 1)).reshape(-1)

model = keras.Sequential([keras.layers.Dense(1),
                          keras.layers.Dense(128, activation = "tanh"),
                          keras.layers.Dense(1, activation = "tanh")])

early_stopping = keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 5)
model.compile(optimizer = 'rmsprop', loss = 'mse')
history = model.fit(X_train, y_train, epochs = 100, callbacks=[early_stopping], 
                    batch_size = 2048, validation_data=(X_test, y_test))

y_hat = scaler_y.inverse_transform(model.predict(X_test)).reshape(-1)
y_test = scaler_y.inverse_transform(y_test).reshape(-1)

f, ax = plt.subplots(figsize = (12, 4))
ax.plot(sorted(scaler_X.inverse_transform(X_test).reshape(-1)), sorted(y_test), color = 'blue', label = 'Real')
ax.plot(sorted(scaler_X.inverse_transform(X_test).reshape(-1)), sorted(y_hat), color = 'orange', label = 'DL')
  • $\begingroup$ Excellent. Rather surprising for me because I have read that relu should (almost) always work. For the moment I do not undersand the divergence with values bigger than 100. I will study your answer and try all your suggestions. I hope I shall be able to fit it beyond 100 :). I supposed that it was not neccesary to do any sort of "feature engineering", as is very frequently asserted. – $\endgroup$ Jul 26 at 13:51
  • 1
    $\begingroup$ For better understanding, inspect tanh figure, it maps $(-\infty, \infty)$ to $(-1, 1)$. Having clarified that, inspect the scaled value of the mass of $R = 100$. scaler_y.transform(np.array([[mass_of_sphere(100)]])) You can simply bypass this limitation by either changing activation function or using a higher range of training values, going for perhaps 500 instead of 120. I just tested with values between (1, 500) and convergence occured at 400. Not an elegant solution but good to have. $\endgroup$
    – crinix
    Jul 26 at 14:07
  • $\begingroup$ I've got a very good fit using "elu" as an activation function. But I was surprised because comparing results (inside the range used) obtained from mass_of_sphere() and model.predict I found enourmous differences. For the moment, I do not identify the error $\endgroup$ Aug 16 at 12:36
  • $\begingroup$ OK maybe a better approach: Scale your data using MinMaxScaler() to range(-1, 1) instead of StandardScaler(). Train your network with tanh activation in last layer again. Don't forget to scale back your prediction. $\endgroup$
    – crinix
    Aug 16 at 20:24

A simpler model seems to be the best option

model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])
model.compile(optimizer='sgd', loss='mean_squared_error')
xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)
ys2 = np.array([4/3*r*np.pi**3 for r in xs])    
model.fit(xs, ys2, epochs=500,validation_split=0.2)

def masa_circulo(x):
    return 4/3*x*np.pi**3

Testing it graphically

x = [x for x in range(1,int(1e6),int(1e3))]
y_masa_circulo = [ masa_circulo(m) for m in x]
y_masa_predicha= [model.predict([m])[0] for m in x]

import matplotlib.pyplot as plt
fig,axes = plt.subplots(1,2)


Graphical comparison function_results vs nn_results

No need of scaling the data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.