Sutton-Barto, page 174.

enter image description here

b successor states are equally likely and in which the error in the initial estimate is 1. The values at the next states are assumed correct, so the expected update reduces the error to zero upon its completion.

The advantage of sample updates shown in Figure 8.7 is probably an underestimate of the real effect. In a real problem, the values of the successor states would be estimates that are themselves updated. By causing estimates to be more accurate sooner, sample updates will have a second advantage in that the values backed up from the successor states will be more accurate. These results suggest that sample updates are likely to be superior to expected updates on problems with large stochastic branching factors and too many states to be solved exactly.

Question: How can Figure 8.7 probably be an underestimate of the real effect? Figure 8.7 is based on the fact that the value of successor states is correct. But in reality instead of these correct values estimated values will be used in updates. Then, it seems to me that Figure 8.7 is probably an overestimation of the real effect.

  • $\begingroup$ What are you reading/interpreting as "the effect" here? $\endgroup$ Commented May 16 at 21:06
  • $\begingroup$ I am very confused what the authors try to mean. It is not clear what they mean by effect but what I understand is effect is used to mean what happens in reality. $\endgroup$ Commented May 17 at 3:21
  • $\begingroup$ In the question, you are asking about (and disagreeing with the author) about "the effect" being smaller or larger. From the last line of this question - "fig 8.7 is probably an overestimate of the effect" - when you use the words "the effect" in that sentence, what value, quantity or quality are you referring to (regardless of what the author is maybe referring to). $\endgroup$ Commented May 17 at 6:14
  • $\begingroup$ my understanding by the word "effect" was that authors meant in reality RMS values in figure 8.7 would be lower when compared to the use of exact values of the next states. I was expecting the opposite of my understanding. $\endgroup$ Commented May 17 at 15:24
  • 1
    $\begingroup$ OK, so you are wrong about what the authors are referring to, but then correct in your reasoning about what would change (for the thing you are thinking of) when going from simplified to real. Could you add your understanding here to the question please, then I can answer $\endgroup$ Commented May 17 at 15:30


You must log in to answer this question.

Browse other questions tagged .