*Disclaimer*: I asked this question 2 days ago in [Cross Validated][1], but it has been left unanswered.

-------------------------

I am trying to better understand how echo state networks work. To see, how fixing the weights of the reservoir of an echo state network impacts the prediction quality of an echo state network, I have conducted a very simple experiment using [ESN layer][1] of the tensorflow-implemented Keras having the two models below:

    model_untrainable = tf.keras.models.Sequential([
        tfa.layers.ESN(units= 1000, spectral_radius=0.99, trainable=False),
        tf.keras.layers.Dense(1, kernel_initializer="lecun_normal")
    ])
    
    model_trainable = tf.keras.models.Sequential([
        tfa.layers.ESN(units= 1000, spectral_radius=0.99),
        tf.keras.layers.Dense(1, kernel_initializer="lecun_normal")
    ])

So, as one sees, the only difference is that in the `model_trainable`, the reservoir's weights are allowed to be updated during training, but in the `model_untrainable`, they are just fixed (by setting `trainable=False`). My hypothesis was that the `model_trainable` should be way better because its settable parameters are more than those of the `model_untrainable`. So, using identical setting for optimizers, loss functions, and regularization and taking [monthly sunspots][2] dataset into account, here are the predictivity results of the cited models.

`model_trainable`:

[![enter image description here][3]][3]

`model_untrainable`:

[![enter image description here][4]][4]

It seems that the `untrainable_model` is almost as good as the `model_trainable`. Why is that the case? In other words, shouldn't the `trainable_model` significantly better than the `untrainable_model`? 


  [1]: https://stats.stackexchange.com/questions/605139/why-do-training-and-fixing-a-reservoir-yield-very-similar-results-in-an-echo-st
  [2]: https://raw.githubusercontent.com/jbrownlee/Datasets/master/monthly-sunspots.csv
  [3]: https://i.sstatic.net/VzndN.jpg
  [4]: https://i.sstatic.net/MGLyo.jpg

  [1]: https://stats.stackexchange.com/questions/605139/why-do-training-and-fixing-a-reservoir-yield-very-similar-results-in-an-echo-st