This topic has been introduced in "Pattern Recognition and Machine Learning, Bishop, 2006", section 5.4.1. I am a bit confused about this method and I have two questions.
Why this method has attracted attention or has been developed? First, we want to compute the Hessian fast, so we try to approximate it in O(W) time, where W is the number of parameters. And then, we see that this matrix most of the time is heavily non-diagonal.
My second question is how can we know whether we can approximate a Hessian or not? Is there a hint/clue in problems?
Thanks in advance!