1
$\begingroup$

From the David Silver's lecture 8: Integrating Learning and Planning - based on Sutton and Barto - he talks about using sample-based planning to use our model to take a sample of a state and then use model-free planning, such as Monte Carlo, etc, to run the trajectory and observe the reward. He goes on to say that this effectively gives us infinite data from only a few actual experiences.

However, if we only experience a handful of true state-action-rewards and then start sampling to learn more then we will surely end up with a skewed result, e.g., If I have 5 experiences but then create 10000 samples (as he says, infinite data). I am aware that as the experience set grows the Central Limit Theorem will come into play and the distribution of experience will more accurately represent the true environment's state-actions-rewards distribution but before this happens is sampled based planning still useful?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
2
$\begingroup$

I am aware that as the experience set grows the Central Limit Theorem will come into play and the distribution of experience will more accurately represent the true environment's state-actions-rewards distribution

I believe here you mean the Law of Large Numbers which states that for a large enough sample ($n \rightarrow \infty$) the sample mean will converge to the true mean. The central limit theorem (CLT) states that if you take the sum/mean of a set of independent random variables then the distribution of this new RV will be approximately normal as $n \rightarrow \infty$.

before this happens is sampled based planning still useful

As you mentioned, if you had only 5 full episodes of experience to choose from, then this likely would not represent true underlying distributions and so the approximations would not be good -- of course, this will depend on how complex your MDP is, if you had a trivially simple one then it may be enough to represent it well. As David Silver says in his lecture, then one of the disadvantages of planning with a model is that you introduce another set of uncertainty, mainly from when you approximate the properties of the model.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Much appreciated. I did mean the Law of Large Numbers, thanks for the correction. So it all seems a trade-off between MDP complexity and the size of experience set. $\endgroup$
    – BlueTurtle
    May 31, 2020 at 13:29
  • $\begingroup$ yes, as you are just learning the model through supervised learning methods, the same rules apply i.e. the complexity of your problem will determine how much data you need to balance the bias/variance trade off $\endgroup$
    – David
    May 31, 2020 at 16:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .