# How can I deal with random weights initialisation when predicting a time-series sine function?

I am training a simple RNN model in keras to predict a time series.

The time series I am considering is just a sine function

The task to solve is the following:

Given a timeseries of 90 time steps, predict the next 10 time steps.

I am generating my training set by sampling from

where $$\epsilon$$ is just a noise term that shifts the series. I have added the noise to have a good training sample.

Question

I train my model for 10 epochs. I notice that sometimes the predictions are actually very good. Some other times, if I retrain the same model with the same dataset, the predictions are very bad. This has to do with the randomness of weights initialisations, but I was hoping that the training algorithm would have found the "best" solution anyway, irrespectively of the initial weights. When I say "best" solution, I mean a class of solutions whose variance is small.

How can I deal with random weights initialisation? Do I need to add some regulation terms to my network?

I know that I can define an ensemble of multiple models in order to cope with weights initialisation. However, this is a very simple problem, and I was expecting to be able to solve it with a single simple model.

Description of the model and code

You can find a Jupyter notebook here .

My training sample will be time series of lenght 90 and the targets will be the value of the series at the 91st step.

I will predict the 10 steps by first predicting the 91st step. Then, I will plug it in the sample. I will select the last 90 terms and predict the 92nd step and so on.

I generate the samples with this code

def generate_series(n_series, n_timesteps, scale_noise=0, scale_shift_origin=10):
"""
Generate random sin series for testing purposes.
"""
series = []
for _ in range(n_series):
noise = np.random.normal(loc=0.0, scale=scale_noise, size=n_timesteps)
noise_origin = np.random.normal(loc=1, scale=scale_shift_origin, size=1)
x = np.linspace(start=0, stop=100, num=n_timesteps)
y = np.sin(0.3*(x + noise_origin)) + noise
series.append(y)
return np.array(series)

keras.utils.set_random_seed(42)

series = generate_series(10000,100)

X_train, y_train = series[:8000,:-10], series[:8000,-10]
X_train = X_train.reshape(8000, -1,1)
y_train = y_train.reshape(8000, -1)

X_test, y_test = series[8000:,:-10], series[8000:,-10]
X_test = X_test.reshape(2000, -1,1)
y_test = y_test.reshape(2000, -1)


The model I am using is very simple, it is a stack of two RNN layers and a Dense layer.

class TwoLSTM:
def __init__(self, n_units_1:int, n_units_2:int):
self.n_units_1 = n_units_1
self.n_units_2 = n_units_2
self.model = None

self.build()

def build(self):
self.model = keras.models.Sequential([
keras.layers.LSTM(self.n_units_1, return_sequences=True, input_shape=[None, 1]),
keras.layers.LSTM(self.n_units_2),
keras.layers.Dense(1)])

def fit(self, train:tuple, test:tuple, epochs:int, batch_size:int=32,verbose=0):
X_train, y_train = train
X_test, y_test = test

callbacks = [
keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, mode='min', restore_best_weights=True)]
history = self.model.fit(X_train, y_train,
epochs=epochs, batch_size=batch_size,
validation_data=(X_test, y_test), callbacks=callbacks,
shuffle=True, verbose=verbose)
return history


I generate the predictions with this code

def generate_predictions(series, model):
X = series.reshape(1,-1,1)
X_new = X[:,:90,:]
for step in range(10):
X_to_use = X_new[:,step:90+step,:]
y_pred = model.predict(X_to_use, verbose=0).reshape(1,-1,1)
X_new = np.concatenate([X_new, y_pred], axis=1)

return X_new


The first training of 10 epochs returns the following prediction of a series in the training set

and the log of the loss function during training are

Then, I train again the same model (by redefining it again; I don't use the best weights of the first run as starting point) and this time I get very good predictions and loss functions

and

• What values are you using for n_units_1 and n_units_2 for those graphs? Nov 16, 2023 at 23:16
• @NeilSlater sorry, I should have written it in the post. I am using n_units_1 = n_units_2 = 1. It is on the jupyter notebook though. Nov 17, 2023 at 6:47

Increasing n_units_1 and n_units_2 to a slightly higher value, say 10, should resolve this provided you are using a normal initialisation routine, such as Keras' default.