I was reading the gradient temporal difference learning version 2(GTD2) from rich Sutton's book page-246. At some point, he expressed the whole expectation using a single sample from the environment. But how a single sample can represent the whole expectation.

I marked this point in this image.

enter image description here

  • 1
    $\begingroup$ This is just a simple SGD algorithm so one sample is enough. $\endgroup$
    – bitWise
    Apr 27 '20 at 15:24

In the, presumably final, printed version the last two equal signs are approximations. This is just because over a large amount of weight updates where you have been sampling the expectation will be approximated by Monte Carlo.

  • $\begingroup$ What you mean by saying expectation will be approximated by Monte Carlo? $\endgroup$ May 9 '20 at 9:01
  • $\begingroup$ You can estimate an expectation by using Monte Carlo methods. An example would be if you had a Normal(0,1) distribution and took enough samples from it then the expectation of a random variable with that normal distribution would be approximated by the sample mean of all your samples. You can do this from any distribution you can sample from - in your example the samples come from repeated interaction with the environment. $\endgroup$ May 10 '20 at 9:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.