Disclaimer: I asked this question 2 days ago in Cross Validated, but it has been left unanswered.
I am trying to better understand how echo state networks work. To see, how fixing the weights of the reservoir of an echo state network impacts the prediction quality of an echo state network, I have conducted a very simple experiment using ESN layer of the tensorflow-implemented Keras having the two models below:
model_untrainable = tf.keras.models.Sequential([ tfa.layers.ESN(units= 1000, spectral_radius=0.99, trainable=False), tf.keras.layers.Dense(1, kernel_initializer="lecun_normal") ]) model_trainable = tf.keras.models.Sequential([ tfa.layers.ESN(units= 1000, spectral_radius=0.99), tf.keras.layers.Dense(1, kernel_initializer="lecun_normal") ])
So, as one sees, the only difference is that in the
model_trainable, the reservoir's weights are allowed to be updated during training, but in the
model_untrainable, they are just fixed (by setting
trainable=False). My hypothesis was that the
model_trainable should be way better because its settable parameters are more than those of the
model_untrainable. So, using identical setting for optimizers, loss functions, and regularization and taking monthly sunspots dataset into account, here are the predictivity results of the cited models.
It seems that the
untrainable_model is almost as good as the
model_trainable. Why is that the case? In other words, shouldn't the
trainable_model significantly better than the