2
$\begingroup$

Slightly generalizing the definition in Jaeger 2001, let's define a reservoir system to be any system of the form

$$h_{t}=f(h_{t-1}, x_t)$$ $$y_t=g(Wh_t)$$

where $f$ and $g$ are fixed and $W$ is a learnable weight matrix. The idea is we feed a sequence of inputs $x_t$ into the system, which has some fixed initial state $h_0$, and thereby generate the sequence of outputs $y_t$. $f$ is fixed (for example, a randomly generated RNN) we can then attempt to learn $W$ in some way in order to get the system to have the behavior that we want.

Now we add the echo state condition: the system has the echo state condition iff for any left-infinite sequence $...x_{-3}, x_{-2}, x_{-1}, x_0$, there is only one sequence of states $h_t$ consistent with this input sequence.

Seen from this perspective, any training procedure that could be applied to an echo state system could be applied to a generic reservoir system. So what do we get out of the echo state condition? Is there some reason to think echo state systems will generalize better, or be more quickly trainable? Jaeger does not seem to attempt to argue in this direction, he just describes how to train an ESN, but as I've said, nothing about these training methods seems to require the echo state property.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The echo state condition states that differences in the input sequence results in separate trajectories of reservoir states.

Which means that when you make the reservoir large enough any difference in the input sequence result in a linearly separable difference in the state space.

This is quite similar to the Kerneltrick in e.g. Support Vector machines where the data becomes linearly separable in the feature space. In reservoir computing the reservoir is in a way a random feature space.

To sum up, the echo state condition ensures that the signal becomes linearly separable.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Can you state this intuition in the form of a theorem? It is not clear to me how you get linear separability just from echo state (or even what "linearly separable" really means in the context of sequence-to-sequence transformations). $\endgroup$
    – Jack M
    Commented Dec 1, 2020 at 16:51
  • $\begingroup$ In this articel (scholarpedia.org/article/Echo_state_network) Jaeger writes: "ESP states that the reservoir will asymptotically wash out any information from initial conditions." This "fading memory property" was formalized by Boyd et al (Fading Memory and the Problem of Approximating Nonlinear Operators with Volterra Series, 1985): for all epsilon there exists delta>0 and T >0 such that |F(v)(t=0)-F(u)(t=0)| < epsilon with |u(t) - v(t)| < delta for all t \in [-T,0]. $\endgroup$
    – claire_fontaine
    Commented Dec 4, 2020 at 17:36
  • $\begingroup$ where F is some time-invariant, causal Operator (the reservoir in our case). $\endgroup$
    – claire_fontaine
    Commented Dec 4, 2020 at 17:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .