I'm reading the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton).

The authors provide the pseudocode of the prioritized sweeping algorithm, but I do not know what is the meaning of Model(s, a). Does it mean that Model(s, a) is the history of rewards gained when we are in state s and the action a is taken?

Does R, S_new = Model(s,a) mean that we should take a random sample from rewards gained in state s and action a is taken?


I think pseudocode was made for tabular case with an assumption of deterministic environment. $Model(s, a)$ would then be a table with information of the next state and reward after taking action $a$ from state $s$. The size of that table would be same as the size of Q table. Because the environment is deterministic you wouldn't take a random sample because there is only one possible transition so you would take the transition remembered in model table.

  • $\begingroup$ Thanks for your response. Do you mean that if we are in state s and take action a, we will be always in the state S_new in the next state? It seems somehow weird! $\endgroup$ – Katatonia Jan 10 '19 at 11:16
  • 1
    $\begingroup$ Yes, it's weird because it was made with an assumption of deterministic environment which is quite optimistic and unreasonable. I never tried it myself but i believe if you have stochastic environment, for tabular case, you could remember all the different transitions that happened and also have a count of how many times those transitions happened, then you could calculate probabilities of those transitions happening from those counts and do planning with that. Maybe you could do sampling like you suggested but based on those probabilities or maybe some kind of dynamic programming updates. $\endgroup$ – Brale Jan 10 '19 at 13:49
  • $\begingroup$ Thanks. Have you ever seen a paper that used the stochastic version? In the deterministic version, does the model use the last visit of next state and reward? $\endgroup$ – Katatonia Jan 10 '19 at 18:28
  • 1
    $\begingroup$ As for the paper, google search has led me to this . I haven't read the paper but it looks like exactly what you are looking for. As for the deterministic environment, yes, model simply uses last observed next state and reward. $\endgroup$ – Brale Jan 10 '19 at 19:11
  • $\begingroup$ Dear @Brale I really appreciate you. $\endgroup$ – Katatonia Jan 11 '19 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.