Which work originally introduced gradient clipping?

Question

The Deep Learning book mentions that it's been used for years but the oldest sources it mentions are from 2012:

A simple type of solution has been in use by practitioners for many years: clipping the gradient. There are different instances of this idea (Mikolov, 2012; Pascanu et al., 2013). One option is to clip the parameter gradient from a mini-batch element-wise (Mikolov, 2012), just before the parameter update. Another is to clip the $||g||$ of the gradient $g$ (Pascanu et al., 2013) just before the parameter update

But I find it hard to believe that the first uses and mentions of gradient clipping are from 2012. Does anyone know the origins of the solution?

Answer 1

Short Answer

Tomas Mikolov's mention of gradient clipping in a single paragraph of his PhD thesis in 2012 is the first appearance in the literature.

Long Answer

The first source (Mikolov, 2012) in the Deep Learning book is Mikolov's PhD thesis and can be found here. The end of section 3.2.2 is where gradient clipping is discussed, only it's called turncating.

... The exploding gradient problem has been described in [4].

A simple solution to the exploding gradient problem is to truncate values of the gradients. In my experiments, I did limit maximum size of gradients of errors that get accumulated in the hidden neurons to be in a range < −15; 15 >. This greatly increases stability of the training, and otherwise it would not be possible to train RNN LMs successfully on large data sets.
...
[4] Y. Bengio, P. Simard, P. Frasconi. Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks, 5, 157-166, 1994.

A search of the referenced paper [4] shows that it does describe the problem as Mikolov said, but it does not present gradient clipping as a solution.

So I had a look at the second source Deep Learning mentioned: On the difficulty of training Recurrent Neural Networks. It directly cites Mikolov as having proposed clipping:

We would make a final note about the approach proposed by Tomas Mikolov in his PhD thesis (Mikolov, 2012) (and implicitly used in the state of the art results on language modelling (Mikolov et al., 2011)). It involves clipping the gradient’s temporal components element-wise (clipping an entry when it exceeds in absolute value a fixed threshold). Clipping has been shown to do well in practice and it forms the backbone of our approach.

I thought about emailing Mikolov to verify that his thesis was the origin of the idea. But then I noticed that he is a co-author of this paper which cites him as proposing it! Though I still wonder if it was commonly used in practice before even though it had not been published.