How can I deal with random weights initialisation when predicting a time-series sine function?

Question

I am training a simple RNN model in keras to predict a time series.

The time series I am considering is just a sine function

The task to solve is the following:

Given a timeseries of 90 time steps, predict the next 10 time steps.

I am generating my training set by sampling from

where $\epsilon$ is just a noise term that shifts the series. I have added the noise to have a good training sample.

Question

I train my model for 10 epochs. I notice that sometimes the predictions are actually very good. Some other times, if I retrain the same model with the same dataset, the predictions are very bad. This has to do with the randomness of weights initialisations, but I was hoping that the training algorithm would have found the "best" solution anyway, irrespectively of the initial weights. When I say "best" solution, I mean a class of solutions whose variance is small.

How can I deal with random weights initialisation? Do I need to add some regulation terms to my network?

I know that I can define an ensemble of multiple models in order to cope with weights initialisation. However, this is a very simple problem, and I was expecting to be able to solve it with a single simple model.

Description of the model and code

You can find a Jupyter notebook here .

My training sample will be time series of lenght 90 and the targets will be the value of the series at the 91st step.

I will predict the 10 steps by first predicting the 91st step. Then, I will plug it in the sample. I will select the last 90 terms and predict the 92nd step and so on.

I generate the samples with this code

def generate_series(n_series, n_timesteps, scale_noise=0, scale_shift_origin=10):
    """
    Generate random sin series for testing purposes.
    """
    series = []
    for _ in range(n_series):
        noise = np.random.normal(loc=0.0, scale=scale_noise, size=n_timesteps)
        noise_origin = np.random.normal(loc=1, scale=scale_shift_origin, size=1)
        x = np.linspace(start=0, stop=100, num=n_timesteps)
        y = np.sin(0.3*(x + noise_origin)) + noise
        series.append(y)
    return np.array(series)

keras.utils.set_random_seed(42)

series = generate_series(10000,100)

X_train, y_train = series[:8000,:-10], series[:8000,-10]
X_train = X_train.reshape(8000, -1,1)
y_train = y_train.reshape(8000, -1)

X_test, y_test = series[8000:,:-10], series[8000:,-10]
X_test = X_test.reshape(2000, -1,1)
y_test = y_test.reshape(2000, -1)

The model I am using is very simple, it is a stack of two RNN layers and a Dense layer.

class TwoLSTM:
    def __init__(self, n_units_1:int, n_units_2:int):
        self.n_units_1 = n_units_1
        self.n_units_2 = n_units_2
        self.model = None
        
        self.build()
        
    def build(self):
        self.model = keras.models.Sequential([
        keras.layers.LSTM(self.n_units_1, return_sequences=True, input_shape=[None, 1]),
        keras.layers.LSTM(self.n_units_2),
        keras.layers.Dense(1)])
        
        self.model.compile(loss="mse", optimizer="adam")
    
    def fit(self, train:tuple, test:tuple, epochs:int, batch_size:int=32,verbose=0):
        X_train, y_train = train
        X_test, y_test = test
        
        callbacks = [
        keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, mode='min', restore_best_weights=True)]
        history = self.model.fit(X_train, y_train, 
                        epochs=epochs, batch_size=batch_size, 
                        validation_data=(X_test, y_test), callbacks=callbacks,
                        shuffle=True, verbose=verbose)
        return history

I generate the predictions with this code

def generate_predictions(series, model):
    X = series.reshape(1,-1,1)
    X_new = X[:,:90,:]
    for step in range(10):
        X_to_use = X_new[:,step:90+step,:]
        y_pred = model.predict(X_to_use, verbose=0).reshape(1,-1,1)
        X_new = np.concatenate([X_new, y_pred], axis=1)
    
    return X_new

The first training of 10 epochs returns the following prediction of a series in the training set

and the log of the loss function during training are

Then, I train again the same model (by redefining it again; I don't use the best weights of the first run as starting point) and this time I get very good predictions and loss functions

and

Answer 1

From comments, you are testing your ideas with a very small neural networks.

The highly variable end result with large dependency on initial conditions is a common result of working with small numbers of neurons. Random initialisation works in part because you can reasonably expect a mean activation value when considered over a layer's output. With a very small number of neurons in any layer, this is less likely to happen. The assumptions behind the statistical analysis that suggest how to properly initialise the parameters in a layer don't hold for very small layers.

Increasing n_units_1 and n_units_2 to a slightly higher value, say 10, should resolve this provided you are using a normal initialisation routine, such as Keras' default.

Another thing that can help is adding some regularisation other than early stopping. A small amount of weight decay (or L2 weight normalisation) may work. Usually you would not apply regularisation to very low number of neurons though, since the low learning capacity of the network means it is already quite restricted. However, one of the issues you may face when increasing the size of the network is that its learning capacity will be much larger than the simple problem you currently are testing it with. So you may need to start addressing over-fitting. Try without it initially though.

There is probably a sweet spot for the layer sizes, where 99%+ of your runs will be stable and accurate over the whole domain, but you won't need any more regularisation.