What does 'clock rate' mean in the context of recurrent neural networks (RNNs)?

Question

I have often encountered the term 'clock rate' when reading literature on recurrent neural networks (RNNs). For example, see this paper. However, I cannot find any explanations for what this means. What does 'clock rate' mean in this context?

Answer 1

The purpose of a clockwork RNN is to help with long term dependencies. Let's say in this case, we have a sentence that starts with "John went to..." and at no point again is John's name mentioned throughout the few paragraphs we are passing to our model.

As mentioned in the paper, the most common method to combat this (at the time) was using an LSTM that stored long term data in it's cell state, or as put in the paper:

[an LSTM] uses a specialized architecture that allows information to be stored in a linear unit called a constant error carousel (CEC) indefinitely

However, this requires a whole heap of extra parameters in order to work (input gate, output gate and forget gate which all require parameters). So proposed was the CW-RNN.

The fundamental idea behind a clockwork RNN is to have "modules" computed periodically. First to clear things up, a timestep is 1 input into the model. So in the case of our example, if we're inputting character by character, timestep 1 is the character "J", timestep 2 is "h" and so on; "o", "n", " ", "w"...

So what is the clock-rate? Well, it's simply how often each module of the CW-RNN is computed. Let's say the hidden layer of the clockwork RNN is split into 8 modules, which I will reference as $M_1, M_2, M_3 ... M_8$, and the associated clock-rates for each of these modules are the powers of 2, so: $1, 2, 4, 8, 16, 32, 64$ and $128$.

Before I continue I want to note a potential discrepancy, if you consider the first input to be timestep 0, then it changes how this executed (all modules would be activated at the first timestep), however if you consider it to be 1 (like I do for this example), only module 1 would be executed (again, in this example).

So we're at timestep 1, ie "J", so we check the timestep against the clock-rates of each module to determine which ones will be executed in the computation of the hidden state and output for this timestep. To do this, we take the mod of the timestep against the time-rate, so: 1 mod 1, 1 mod 2, 1 mod 4 ... 1 mod 128 and if it equals 0 (basically, is the timestep a multiple of the time-rate) then that module will be executed at this timestep. So in this case $M_1$ will be executed. When we input "h" at timestep 2, $M_1$ and $M_2$ will both be executed, and will equally contribute to the hidden state (ie, they will be added together) and the output.

By performing calculations this way, each module can all simultaneously be responsible for information over different time periods, for example $M_8$ which is only executed every 128 timesteps will be responsible for very long term dependencies, but $M_1$ will cover short term dependencies.

So in basic terms, clock-rate refers to how often (what timestep interval) a given module of a Clockwork RNN is computed

Answer 2

It seems this paper defines a "clock period ${T}_n$" that it uses to express the topology of the network: "Each module is internally fully-interconnected, but the recurrent connections from module j to module i exists only if the period $T_i$ is smaller than period $T_j$.".

This definition is, however, only in this paper, as far as I know. It wouldn't make sense to have a "clock period" (or "clock rate") on other, more self-similar, RNNs. However "time t" is something that is defined to express the amount of times the RNN has recursed, "t times".