Disclaimer: I asked this question 2 days ago in Cross Validated, but it has been left unanswered.

I am trying to better understand how echo state networks work. To see, how fixing the weights of the reservoir of an echo state network impacts the prediction quality of an echo state network, I have conducted a very simple experiment using ESN layer of the tensorflow-implemented Keras having the two models below:

model_untrainable = tf.keras.models.Sequential([
    tfa.layers.ESN(units= 1000, spectral_radius=0.99, trainable=False),
    tf.keras.layers.Dense(1, kernel_initializer="lecun_normal")
])

model_trainable = tf.keras.models.Sequential([
    tfa.layers.ESN(units= 1000, spectral_radius=0.99),
    tf.keras.layers.Dense(1, kernel_initializer="lecun_normal")
])

So, as one sees, the only difference is that in the model_trainable, the reservoir's weights are allowed to be updated during training, but in the model_untrainable, they are just fixed (by setting trainable=False). My hypothesis was that the model_trainable should be way better because its settable parameters are more than those of the model_untrainable. So, using identical setting for optimizers, loss functions, and regularization and taking monthly sunspots dataset into account, here are the predictivity results of the cited models.

model_trainable:

model_untrainable:

It seems that the untrainable_model is almost as good as the model_trainable. Why is that the case? In other words, shouldn't the trainable_model significantly better than the untrainable_model?

Disclaimer: I asked this question 2 days ago in Cross Validated, but it has been left unanswered.

I am trying to better understand how echo state networks work. To see, how fixing the weights of the reservoir of an echo state network impacts the prediction quality of an echo state network, I have conducted a very simple experiment using ESN layer of the tensorflow-implemented Keras having the two models below:

model_untrainable = tf.keras.models.Sequential([
    tfa.layers.ESN(units= 1000, spectral_radius=0.99, trainable=False),
    tf.keras.layers.Dense(1, kernel_initializer="lecun_normal")
])

model_trainable = tf.keras.models.Sequential([
    tfa.layers.ESN(units= 1000, spectral_radius=0.99),
    tf.keras.layers.Dense(1, kernel_initializer="lecun_normal")
])

So, as one sees, the only difference is that in the model_trainable, the reservoir's weights are allowed to be updated during training, but in the model_untrainable, they are just fixed (by setting trainable=False). My hypothesis was that the model_trainable should be way better because its settable parameters are more than those of the model_untrainable. So, using identical setting for optimizers, loss functions, and regularization and taking monthly sunspots dataset into account, here are the predictivity results of the cited models.

model_trainable:

model_untrainable:

It seems that the untrainable_model is almost as good as the model_trainable. Why is that the case? In other words, shouldn't the trainable_model significantly better than the untrainable_model?