2
$\begingroup$

I came across this graph in David Silver's youtube lecture and Sutton's book on reinforcement learning.

Can anyone help me understand the graph? From the graph, for 10000 episodes what i see is that when we don't have a usable ace we always lose the game except if the sum is 20 or 21. But in case if we have a usable ace there are chances to win when our sum is below 20. I don't know how this is possible.

enter image description here

$\endgroup$
  • 4
    $\begingroup$ Hi, it would be useful if you could edit your question and add some details about what it is you don't understand. $\endgroup$ – Oliver Mason Feb 3 at 10:01
  • $\begingroup$ @OliverMason i have updated the question. Thanks for your comments. $\endgroup$ – Bhuwan Bhatt Feb 5 at 10:45
2
$\begingroup$

The left hand graphs are showing you the estimated value function from using Monte Carlo evaluation, after 10,000 episodes. They give a sense of what your value table will look like before convergence.

In the case of upper "usable ace" chart, the estimates are still showing a lot of inaccuracy due to variance in the data. This is for two main reasons:

  • The probability of getting a usable ace at the start is a fraction of all (around 15%), so the number of samples used to build the chart is lower.

  • There are more variations in play when there is a usable ace, due to the extra flexibility allowed by it, so the end result also varies more, requiring more samples to converge.

In addition, if you are looking at the bottom edge of the chart, this represents the player starting with two aces. If you pick on one of the high points (dealer shows 4), then that also reduces the probability of seeing that particular state. So you are looking at a sample size of typically 4-5, but maybe in this case just one sample or maybe two, which the player then happened to go on to win, even though the odds made it unlikely. There is always some chance of winning and dealer showing 4 is a bad start for the dealer, who has a good chance of going bust provided the player does not.

If it hadn't happened this time for the "two aces + dealer showing 4" state, it may have happened for "two aces + dealer showing 5" state. That's due to the nature of random sampling - if you have hundreds of states to sample, then a few of them are going to behave like outliers purely by chance until you have taken enough samples.

In short, 10,000 randomly sampled games are nowhere near enough to reduce error bounds on the value estimates to reasonable numbers for special cases such as starting with two aces. However, you can see in the 10,000 samples charts the beginnings of convergence, especially elsewhere in the chart.

From the graph, for 10000 episodes what i see is that when we don't have a usable ace we always lose the game except if the sum is 20 or 21.

Actually you don't see that, the expected result is not quite -1.0, but a little higher. So that means there is still a chance to win. Under this player policy, the worst chance is no usable ace and score 19, because the policy will be to "hit" and need an Ace or 2 card just to stay in the game. Even then , the value is not quite as low as -1.0, but more like -0.9

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.