I was reading the gradient temporal difference learning version 2(GTD2) from rich Sutton's book page-246. At some point, he expressed the whole expectation using a single sample from the environment. But how a single sample can represent the whole expectation.

I marked this point in this image.

enter image description here

  • 1
    $\begingroup$ This is just a simple SGD algorithm so one sample is enough. $\endgroup$
    – bitWise
    Apr 27, 2020 at 15:24

1 Answer 1


In the, presumably final, printed version the last two equal signs are approximations. This is just because over a large amount of weight updates where you have been sampling the expectation will be approximated by Monte Carlo.

  • $\begingroup$ What you mean by saying expectation will be approximated by Monte Carlo? $\endgroup$ May 9, 2020 at 9:01
  • $\begingroup$ You can estimate an expectation by using Monte Carlo methods. An example would be if you had a Normal(0,1) distribution and took enough samples from it then the expectation of a random variable with that normal distribution would be approximated by the sample mean of all your samples. You can do this from any distribution you can sample from - in your example the samples come from repeated interaction with the environment. $\endgroup$
    – David
    May 10, 2020 at 9:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .