# How DynaQ behaves in stochastic world in comparison with other reinforcement learning algorithms?

I came across of implementations of a bunch of algorithms on stochastic windy gridworld. This is the graph comparing their performance:

So clearly, it seems that DynaQ performs better than all other algorithms. Also, as $$n$$ goes on increasing, the performance of DynaQ also goes on increasing. But somehow I feel this contradicts with what Sutton and Barto says in the book (section 8.3):

Models may be incorrect because the environment is stochastic and only a limited number of samples have been observed, … When the model is incorrect, the planning process is likely to compute a suboptimal policy. So, I was wondering why DynaQ does not perform poorly than Q learning for "stochastic" windy gridworld.

Q1. Is it that the authors did not want to compare DynaQ with Q learning, but say in general, with the above quoted statement? (this might be the case as the corresponding paragraph does not mention Q learning at all)

Q2. Can no amount of stochasticity (in any environment, not just windy gridworld) make DynaQ perform poorer than Q-learning? (for windy grid world, we may consider making all column stochastic instead of just windy columns or wind may take us to five different states instead of three)

Q3. The quoted sentence says "only a limited number of samples have been observed". What does it refer to by "samples"? Inner loop or outer loop (f)? My logic is that it must be an outer loop as for getting correct T and R values for a given state, we must pass through it several times. That is, we should have completed a large number of episodes. But, then the term "sample" is used in the context of model learning (Loop (f)). So is my logic incorrect?

Q4. Also, for the stochastic world, which has the incorrect model at some point in time of DynaQ execution, does larger $$n$$ leads to a more incorrect model or a less incorrect model?

Q3 and Q4 arose because I am not able to visualize how (correct/incorrect) policies will result in a stochastic world, say for halfway through the second episode, as given in the book for the deterministic world:

PS: The main question is as it appears in the title. I have formed four subquestions so that answerer can opt to focus answer on them individually to answer question in the title from all possible aspects.

• In any case, I noticed that you also asked this question on Stats SE. Try to avoid cross-duplication.
– nbro
Feb 17 at 14:32