1
$\begingroup$

I am trying to implement a tabular-based GLIE Monte-Carlo learning algorithm. So I repeat n times:

  1. create observations using my previous policy $\pi_{n-1}(s)$
  2. update my state-action values using the observations generated in 1 with the monte-carlo update rule: $Q_n(s_t,a_t)= Q_n(s_t,a_t)+1/N(s_t,a_t)\times(G_t-Q_n(S_t,a_t))$
  3. update my policy to $\pi_{n}$ using epsilon-geedy improvement with $\epsilon=1/(n+1)$.

In step 2 I need to decide for an initial estimate $\tilde{Q}_n$. Is it a decent option to use $\tilde{Q}_n=Q_{n-1}$?

$\endgroup$
1
$\begingroup$

In step 2 I need to decide for an initial estimate $\tilde{Q}_n$. Is it a decent option to use $\tilde{Q}_n=Q_{n-1}$?

Yes, this is a common choice. It's actually common to update the table for $\tilde{Q}$ in place, without any separate initialisation per step. The separate phases of estimation and policy improvement are easier to analyse for theoretical correctness, but in practice updates made in place can be faster because new information is used as soon as it is available.

Depending on how the policy was changed, and how accurate the previous estimate was, this could place the estimates closer convergence for the next step. Often the previous estimates will be closer to the new targets than any fixed or random initialisation scheme you could set up.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you! However I am not sure whether I understand the term "in place" does this mean that I improve my policy after every Monte-Carlo Update? $\endgroup$ – Sebastian Nov 10 '19 at 11:15
  • $\begingroup$ I mean that you only have one Q table, and overwrite values as you calculate them. So no need to initialise a new table on each time step $\endgroup$ – Neil Slater Nov 10 '19 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.