Not able to find a good fit for a simple function with neural networks

Question

I have been trying to adjust a neural network to a simple function: the mass of an sphere. I have tried with different architectures, for example, a single hidden layer and two hidden layers, always with 128 neurons each, and training them for 5000 epochs. The code is the usual one. Just in case, I publish one of them

model = keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])
                        ,keras.layers.Dense(128, activation="relu")
                        ,keras.layers.Dense(1, activation="relu")])
model.compile(optimizer='rmsprop', loss='mse', metrics=['mae'])
history = model.fit(x, y, validation_split=0.2, epochs=5000)

The results are shown in the graphs.

I suspect that I am making an error somewhere, because I have seen that deep learning is able to match complex functions with much less epochs. I shall appreciate any hint to fix this problem and obtain a good fit with the deep learning function.

In order to make it clear I post the graph's code.

rs =[x for x in range(20)]
def masas_circulo(x):
    masas_circulos =[]
    rs =[r for r in range(x)]
    for r in rs:
        masas_circulos.append(model.predict([r])[0][0])

   return masas_circulos

 masas_circulos = masas_circulo(20) 
 masas_circulos
 esferas = [4/3*np.pi*r**3 for r in range(20)]
 import matplotlib.pyplot as plt
 plt.plot(rs,masas_circulos,label="DL")
 plt.plot(rs,esferas,label="Real");
 plt.title("Mass of an sphere.\nDL (1hl,128 n,5000 e) vs ground_truth")
 plt.xlabel("Radius")
 plt.ylabel("Sphere")
 plt.legend();

Answer 1

You're trying to learn a cubic function that explodes in values and your issue is scaling. I have been able to learn a better approximation by scaling data and using tanh as activation function.

Code and result are as below:

Convergence around X=100 happens because of tanh activation. Relu will not work better because of negative values that is the result of scaling. You can try playing with Leaky Relu activation and various alpha values.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

def mass_of_sphere(R):
    return (4/3) * np.pi * (R**3)

X = np.linspace(1, 120, 500000)
y = [mass_of_sphere(x) for x in X]

X = np.array(X).reshape(-1, 1)
y = np.array(y)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)

scaler_X = StandardScaler()
X_train = scaler_X.fit_transform(X_train)
X_test = scaler_X.transform(X_test)

scaler_y = StandardScaler()
y_train = scaler_y.fit_transform(y_train.reshape(-1, 1))
y_test = scaler_y.transform(y_test.reshape(-1, 1)).reshape(-1)

model = keras.Sequential([keras.layers.Dense(1),
                          keras.layers.Dense(128, activation = "tanh"),
                          keras.layers.Dense(1, activation = "tanh")])

early_stopping = keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 5)
model.compile(optimizer = 'rmsprop', loss = 'mse')
history = model.fit(X_train, y_train, epochs = 100, callbacks=[early_stopping], 
                    batch_size = 2048, validation_data=(X_test, y_test))

y_hat = scaler_y.inverse_transform(model.predict(X_test)).reshape(-1)
y_test = scaler_y.inverse_transform(y_test).reshape(-1)

f, ax = plt.subplots(figsize = (12, 4))
ax.plot(sorted(scaler_X.inverse_transform(X_test).reshape(-1)), sorted(y_test), color = 'blue', label = 'Real')
ax.plot(sorted(scaler_X.inverse_transform(X_test).reshape(-1)), sorted(y_hat), color = 'orange', label = 'DL')
ax.legend()

Answer 2

A simpler model seems to be the best option

model = tf.keras.Sequential([keras.layers.Dense(units=1, input_shape=[1])])
model.compile(optimizer='sgd', loss='mean_squared_error')
xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)
ys2 = np.array([4/3*r*np.pi**3 for r in xs])    
model.fit(xs, ys2, epochs=500,validation_split=0.2)

def masa_circulo(x):
    return 4/3*x*np.pi**3

Testing it graphically

x = [x for x in range(1,int(1e6),int(1e3))]
y_masa_circulo = [ masa_circulo(m) for m in x]
y_masa_predicha= [model.predict([m])[0] for m in x]

import matplotlib.pyplot as plt
fig,axes = plt.subplots(1,2)
axes[0].plot(y_masa_circulo);
axes[0].set_title("y_masa_circulo")
axes[0].set_ylabel("y_masa_circulo")
axes[0].set_xlabel("Radio")

axes[1].plot(y_masa_predicha);
plt.title("y_masa_circulo")
plt.title("y_masa_predicha");
axes[1].set_ylabel("y_masa_predicha");
axes[1].set_xlabel("Radio");

No need of scaling the data.