3
$\begingroup$

I am trying to implement a tabular-based GLIE Monte-Carlo learning algorithm. So I repeat n times:

  1. create observations using my previous policy $\pi_{n-1}(s)$
  2. update my state-action values using the observations generated in 1 with the monte-carlo update rule: $Q_n(s_t,a_t)= Q_n(s_t,a_t)+1/N(s_t,a_t)\times(G_t-Q_n(S_t,a_t))$
  3. update my policy to $\pi_{n}$ using epsilon-geedy improvement with $\epsilon=1/(n+1)$.

In step 2 I need to decide for an initial estimate $\tilde{Q}_n$. Is it a decent option to use $\tilde{Q}_n=Q_{n-1}$?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In step 2 I need to decide for an initial estimate $\tilde{Q}_n$. Is it a decent option to use $\tilde{Q}_n=Q_{n-1}$?

Yes, this is a common choice. It's actually common to update the table for $\tilde{Q}$ in place, without any separate initialisation per step. The separate phases of estimation and policy improvement are easier to analyse for theoretical correctness, but in practice updates made in place can be faster because new information is used as soon as it is available.

Depending on how the policy was changed, and how accurate the previous estimate was, this could place the estimates closer convergence for the next step. Often the previous estimates will be closer to the new targets than any fixed or random initialisation scheme you could set up.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you! However I am not sure whether I understand the term "in place" does this mean that I improve my policy after every Monte-Carlo Update? $\endgroup$
    – Sebastian
    Commented Nov 10, 2019 at 11:15
  • $\begingroup$ I mean that you only have one Q table, and overwrite values as you calculate them. So no need to initialise a new table on each time step $\endgroup$
    – Neil Slater
    Commented Nov 10, 2019 at 14:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .