I'm new to ML / Reinforcement Learning.

I'm looking at source from https://github.com/openai/gym/blob/master/gym/envs/toy_text/blackjack.py

where reward is calculated with

reward = cmp(score(self.player), score(self.dealer))

I'm just curious why its ok to calculate reward based on hidden state? A player only sees the dealers first card.

  • 1
    $\begingroup$ The reward is just a number. It tells nothing more than that. You, as a teacher or programmer, can model the reward function as you well think. $\endgroup$ – nbro Nov 22 '18 at 9:29

The code you reference is not part of the learning agent. It is part of:

class BlackjackEnv(gym.Env):

If, as in this case, the environment is provided entirely by software simulation, it is absolutely necessary for it to include a full working model of all state transitions and rewards. That is independent of whether any hidden state makes the problem harder.

I'm just curious why its ok to calculate reward based on hidden state?

In the case of Blackjack, this can be treated not as hidden state that would affect the outcome if only known, but as randomness in the environment over which the agent has no control. Critically, the dealer has no options to behave differently depending on the unknown card, and the dealer's eventual score is entirely unaffected by the player's earlier choices.

It is a subtle difference. If you applied the same environment rules to Poker, where an opponent could behave differently depending on this hidden knowledge, then a simple MDP model is not enough theory to result in an optimal solution. In that case you would need to look into Partially Observable MDPs (POMDPs). Note this would not affect reward calculation in the environment, just the choices of which agent type to use. If you are just learning RL, you probably don't know of any algorithms that could solve this yet.

In practice, a lot of problems are somewhere between a classic MDP and a POMDP - they contain elements which, if the agent could know them, may allow it to achieve a higher expected reward. In many cases though, these elements can either be treated as random (as here in Blackjack) and thus the system is still theoretically a MDP, or they have a very small effect on the optimal policy, so can be ignored for practical purposes (e.g. think of all the physical details in a real cart pole balancing system - friction, temperature, flexing motions etc).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.