1
$\begingroup$

I have often encountered the term 'clock rate' when reading literature on recurrent neural networks (RNNs). For example, see this paper. However, I cannot find any explanations for what this means. What does 'clock rate' mean in this context?

$\endgroup$
1
+50
$\begingroup$

The purpose of a clockwork RNN is to help with long term dependencies. Let's say in this case, we have a sentence that starts with "John went to..." and at no point again is John's name mentioned throughout the few paragraphs we are passing to our model.

As mentioned in the paper, the most common method to combat this (at the time) was using an LSTM that stored long term data in it's cell state, or as put in the paper:

[an LSTM] uses a specialized architecture that allows information to be stored in a linear unit called a constant error carousel (CEC) indefinitely

However, this requires a whole heap of extra parameters in order to work (input gate, output gate and forget gate which all require parameters). So proposed was the CW-RNN.

The fundamental idea behind a clockwork RNN is to have "modules" computed periodically. First to clear things up, a timestep is 1 input into the model. So in the case of our example, if we're inputting character by character, timestep 1 is the character "J", timestep 2 is "h" and so on; "o", "n", " ", "w"...

So what is the clock-rate? Well, it's simply how often each module of the CW-RNN is computed. Let's say the hidden layer of the clockwork RNN is split into 8 modules, which I will reference as $M_1, M_2, M_3 ... M_8$, and the associated clock-rates for each of these modules are the powers of 2, so: $1, 2, 4, 8, 16, 32, 64$ and $128$.

Before I continue I want to note a potential discrepancy, if you consider the first input to be timestep 0, then it changes how this executed (all modules would be activated at the first timestep), however if you consider it to be 1 (like I do for this example), only module 1 would be executed (again, in this example).

So we're at timestep 1, ie "J", so we check the timestep against the clock-rates of each module to determine which ones will be executed in the computation of the hidden state and output for this timestep. To do this, we take the mod of the timestep against the time-rate, so: 1 mod 1, 1 mod 2, 1 mod 4 ... 1 mod 128 and if it equals 0 (basically, is the timestep a multiple of the time-rate) then that module will be executed at this timestep. So in this case $M_1$ will be executed. When we input "h" at timestep 2, $M_1$ and $M_2$ will both be executed, and will equally contribute to the hidden state (ie, they will be added together) and the output.

By performing calculations this way, each module can all simultaneously be responsible for information over different time periods, for example $M_8$ which is only executed every 128 timesteps will be responsible for very long term dependencies, but $M_1$ will cover short term dependencies.

So in basic terms, clock-rate refers to how often (what timestep interval) a given module of a Clockwork RNN is computed

$\endgroup$
2
  • $\begingroup$ Thanks for the answer. So does a CW-RNN actually produce superior results to the LSTM (for this purpose of 'long term memory')? $\endgroup$ Apr 16 at 15:28
  • 1
    $\begingroup$ According to the paper, yes it does "CW-RNNs beat both RNN and LSTM networks by a considerable margin of 8-20% on average" for classification and " CW-RNNs give by far the best results, with the smallest one being roughly on par with the second-biggest LSTM network" for sequence generation $\endgroup$
    – Recessive
    Apr 17 at 2:41
1
$\begingroup$

It seems this paper defines a "clock period ${T}_n$" that it uses to express the topology of the network: "Each module is internally fully-interconnected, but the recurrent connections from module j to module i exists only if the period $T_i$ is smaller than period $T_j$.".

This definition is, however, only in this paper, as far as I know. It wouldn't make sense to have a "clock period" (or "clock rate") on other, more self-similar, RNNs. However "time t" is something that is defined to express the amount of times the RNN has recursed, "t times".

$\endgroup$
2
  • $\begingroup$ Thanks for the answer. So what exactly does "clock rate" mean in the context of the linked paper? $\endgroup$ Apr 10 at 19:46
  • $\begingroup$ @ThePointer "clock rate" refers on how modules process data. The idea of the paper is to compute information at different "rates", for this, it uses modules, the modules are connected to others according to the module's clock period. This creates a network where information is processed by each module at different rates. Quote: "Each module $i$ is evaluated only every $T_i$-th time step", this would be a 'rate' of computation of incoming data. $\endgroup$ Apr 10 at 23:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.