2
$\begingroup$

Disclaimer: I asked this question 2 days ago in Cross Validated, but it has been left unanswered.

I am trying to better understand how echo state networks work. To see, how fixing the weights of the reservoir of an echo state network impacts the prediction quality of an echo state network, I have conducted a very simple experiment using ESN layer of the tensorflow-implemented Keras having the two models below:

model_untrainable = tf.keras.models.Sequential([
    tfa.layers.ESN(units= 1000, spectral_radius=0.99, trainable=False),
    tf.keras.layers.Dense(1, kernel_initializer="lecun_normal")
])

model_trainable = tf.keras.models.Sequential([
    tfa.layers.ESN(units= 1000, spectral_radius=0.99),
    tf.keras.layers.Dense(1, kernel_initializer="lecun_normal")
])

So, as one sees, the only difference is that in the model_trainable, the reservoir's weights are allowed to be updated during training, but in the model_untrainable, they are just fixed (by setting trainable=False). My hypothesis was that the model_trainable should be way better because its settable parameters are more than those of the model_untrainable. So, using identical setting for optimizers, loss functions, and regularization and taking monthly sunspots dataset into account, here are the predictivity results of the cited models.

model_trainable:

enter image description here

model_untrainable:

enter image description here

It seems that the untrainable_model is almost as good as the model_trainable. Why is that the case? In other words, shouldn't the trainable_model significantly better than the untrainable_model?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

trainable = False is a hardcoded setting for each ESN layer in esn_cell.py. So, regardless of what one sets in any usage of ESN, its corresponding reservoir cannot be trained.

Improve this answer
$\endgroup$

You must log in to answer this question.