Is the distribution of state-action pairs from sample based planning accurate for small experience sets?

From the David Silver's lecture 8: Integrating Learning and Planning - based on Sutton and Barto - he talks about using sample-based planning to use our model to take a sample of a state and then use model-free planning, such as Monte Carlo, etc, to run the trajectory and observe the reward. He goes on to say that this effectively gives us infinite data from only a few actual experiences.

However, if we only experience a handful of true state-action-rewards and then start sampling to learn more then we will surely end up with a skewed result, e.g., If I have 5 experiences but then create 10000 samples (as he says, infinite data). I am aware that as the experience set grows the Central Limit Theorem will come into play and the distribution of experience will more accurately represent the true environment's state-actions-rewards distribution but before this happens is sampled based planning still useful?

I believe here you mean the Law of Large Numbers which states that for a large enough sample ($$n \rightarrow \infty$$) the sample mean will converge to the true mean. The central limit theorem (CLT) states that if you take the sum/mean of a set of independent random variables then the distribution of this new RV will be approximately normal as $$n \rightarrow \infty$$.