3
$\begingroup$

I'm reading the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton).

The authors provide the pseudocode of the prioritized sweeping algorithm, but I do not know what is the meaning of Model(s, a). Does it mean that Model(s, a) is the history of rewards gained when we are in state s and the action a is taken?

Does R, S_new = Model(s,a) mean that we should take a random sample from rewards gained in state s and action a is taken?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

I think pseudocode was made for tabular case with an assumption of deterministic environment. $Model(s, a)$ would then be a table with information of the next state and reward after taking action $a$ from state $s$. The size of that table would be same as the size of Q table. Because the environment is deterministic you wouldn't take a random sample because there is only one possible transition so you would take the transition remembered in model table.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for your response. Do you mean that if we are in state s and take action a, we will be always in the state S_new in the next state? It seems somehow weird! $\endgroup$
    – Amin
    Commented Jan 10, 2019 at 11:16
  • 1
    $\begingroup$ Yes, it's weird because it was made with an assumption of deterministic environment which is quite optimistic and unreasonable. I never tried it myself but i believe if you have stochastic environment, for tabular case, you could remember all the different transitions that happened and also have a count of how many times those transitions happened, then you could calculate probabilities of those transitions happening from those counts and do planning with that. Maybe you could do sampling like you suggested but based on those probabilities or maybe some kind of dynamic programming updates. $\endgroup$
    – Brale
    Commented Jan 10, 2019 at 13:49
  • $\begingroup$ Thanks. Have you ever seen a paper that used the stochastic version? In the deterministic version, does the model use the last visit of next state and reward? $\endgroup$
    – Amin
    Commented Jan 10, 2019 at 18:28
  • 1
    $\begingroup$ As for the paper, google search has led me to this . I haven't read the paper but it looks like exactly what you are looking for. As for the deterministic environment, yes, model simply uses last observed next state and reward. $\endgroup$
    – Brale
    Commented Jan 10, 2019 at 19:11
  • $\begingroup$ Dear @Brale I really appreciate you. $\endgroup$
    – Amin
    Commented Jan 11, 2019 at 14:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .