3
$\begingroup$

Monte-Carlo Tree Search was the method used for AlphaGo my understanding is: it would randomly search the state space of possible moves where the probability of choosing a move was proportional to the perceived Value of the resulting state (this rollout data was then used to train the value function further).

This seems very similar to MCMC to me. Except that it includes a mechanism for observing ground truth likelihood (winners/losers in GO) & updating the likelihood function (aka Value function) in the process.

Is this true? And to what degree in general have Bayesian methods been used in RL? For example have people tried using one of the various specialized MCMC algorithms to improve Monte-Carlo tree search?

Sorry if the question is open-ended/vague.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

There is an element in MCTS, in which you select to explore "promising" directions. You also give a chance for branches that were not explored enough yet. So, you can use for that Bayesian methods (to keep and maintain the distribution of the expected rewards).

Improve this answer
$\endgroup$
2
  • $\begingroup$ I'm curious if you know if anyone has tried using MCMC for this? $\endgroup$
    – profPlum
    Jul 7 at 3:19
  • $\begingroup$ AlphaGo as far as I can tell at this moment, is basically MCTS + maintaining and updating a value function and using it for making decision where to spend the next iterations. The vanilla MCTS makes use of Upper Confidence Bound (UCB), which is a handy / simple surrogate for a distribution. $\endgroup$
    – Oren
    Jul 8 at 17:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .