3
$\begingroup$

I frequently see people setting deterministic = True while testing an RL algorithm. But is this the right approach? For instance, what happens if the agent plays rock, paper, and scissors? In this case, as per game theory, a stochastic (random) policy is required (as per my understanding)

Edit - Let me clarify my understanding of stochastic and deterministic policies. A deterministic policy will always select an action given some state. The stochastic policy will sample the action given a state. Therefore in the latter case, an optimal policy for Rock, Paper and Scissor will always choose an action with probability of 0.33.

$\endgroup$
1
  • 1
    $\begingroup$ It's worth noting deterministic policy doesn't necessarily mean it always select a unique action given some state, it's possible in finite MDP at some state several actions equally solves the Bellman optimality equation, thus the greedy deterministic optimal policy can randomly follow any such action. $\endgroup$
    – cinch
    Feb 18, 2023 at 5:22

1 Answer 1

4
$\begingroup$

Is it always a good idea to use deterministic policies during testing?

No, it's not, and you correctly point out that in certain scenarios the optimal policy is stochastic.

Specifically, the optimal policy in Rock-Paper-Scissors is a stochastic policy. However, this game can't really be represented as a stationary MDP, but as a Markov Game with 2 players. If you wanted to represent it as an MDP (Markov Game with 1 player), then it might not be stationary, i.e. the dynamics could change.

Having said that, there's a known result in the theory of MDPs that says that there's at least one optimal policy that is stationary and deterministic, but there are other environments and models other than finite MDPs. You also have POMDPs, Markov Games, etc., which are generalisations of finite MDPs to different scenarios (partial observability of the state and multiple players, respectively).

So, when should you use deterministic policies? Well, most likely, when you are sure that your environment is a finite MDP. It's possible that, when they use deterministic policies, people assume the environment can be represented as an MDP, while, in reality, maybe it's not. Then, of course, in practice, you lose all theoretical guarantees.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .