Consider a very simple problem, which is to find the maximum value out of a list of 5 numbers between 0 and 1. This is obviously trivial, but serves as a good example for a real-world problem I'm facing.
One could attempt to train an MLP to solve this problem using randomly generated input data. The following code is an example in Keras/Tensorflow. It generates 1000000 random examples (A large number, so we can reduce overfitting as a factor). It also applies some simple techniques like LR decay and early stopping to optimize training.
import tensorflow as tf # Create a dataset where the y value is the largest of 5 random x values. x = tf.random.uniform(shape=(1000000, 5), minval=0.0, maxval=1.0) y = tf.reduce_max(x, axis=-1) dataset = tf.data.Dataset.from_tensor_slices( (x, y) ).shuffle(1000000).batch(1024).prefetch(1) # See the post for discussion of hyperparameter optimization units = 400 depth = 5 initial_learning_rate = 0.001 model = tf.keras.models.Sequential() for i in range(depth): model.add(tf.keras.layers.Dense(units, activation="relu")) model.add(tf.keras.layers.Dense(1)) model.compile( optimizer=tf.keras.optimizers.Adam(initial_learning_rate), loss=tf.keras.losses.MeanAbsolutePercentageError(), ) stopping_callback = tf.keras.callbacks.EarlyStopping(monitor="loss", patience=5) reduce_lr_callback = tf.keras.callbacks.ReduceLROnPlateau(monitor="loss", factor=0.75, patience=2) model.fit(dataset, epochs=1000)
The metric we care about is the mean absolute percentage error. When we run the code, the model trains successfully. But very importantly, the loss does not ever go below ~0.2 with these hyperparameters.
0.2% error sounds pretty good, but it's actually unacceptable for my use case. I need much more precision, as in practice the model is used in a situation where any errors are magnified.
At this point, you may want to suggest hyperparameter tuning, but I assure you I have done an extensive amount of hyperparameter tuning on this and similar problems.
The model is clearly underfitting, and it is possible to decrease the error by increasing the number of parameters in the network. Empirically, I've noticed that increasing the number of parameters (and number of examples) by an order of magnitude can reduce the error by a similar order of magnitude.
However, intuitively this just feels wrong. It is not practical to have a network with millions, or even tens of millions of parameters, just to regress such a simple function. It feels like there should be an architecture that can learn a simple function much more efficiently, but I have not been able to find any reference to such an alternative architecture.
Again, of course this particular example is trivial, but hopefully you can imagine a real-world analogue where an unknown (more complex, but similar) function is expressed by a similar dataset. And in such a real-world example, the unacceptable error still persists (and is much more pronounced, for reasons that I don't fully understand).
Does anyone have any idea what's going on here? Am I missing something? How can I improve the performance to a place that would be acceptable?