# Tag Info

7

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? No. An optimal policy is generally deterministic unless: Important state information is missing (a POMDP). For example, in a map where the agent is not allowed to know its exact location or remember ...

7

A deterministic policy is a function of the form $\pi_{\mathbb{d}}: S \rightarrow A$, that is, a function from the set of states of the environment, $S$, to the set of actions, $A$. The subscript $_{\mathbb{d}}$ only indicates that this is a ${\mathbb{d}}$eterministic policy. For example, in a grid world, the set of states of the environment, $S$, is ...

5

I would say no. For example, consider the multi-armed bandit problem. So, you have $n$ arms which all have a probability of giving you a reward (1 point, for example), $p_i$, $i$ being between 1 and $n$. This is a simple stochastic environment: this is a one state environment, but it is still an environment. But obviously the optimal policy is to choose ...

5

Did AlphaGo and AlphaGo [Zero] play 100 repetitions of the same sequence of boards, or were there 100 different games? There were 100 different games. You can view some example games between AlphaGo [Lee] and AlphaGo Zero here. They are clearly all different. This statement in the question shows a misunderstanding: My understanding of AlphaGo and AlphaGo [...

3

This update rule can still be applied in the continuous domain. As pointed out in the comments, suppose we are parameterising our policy using a Gaussian distribution, where our neural networks take as input the state we are in and output the parameters of a Gaussian distribution, the mean and the standard deviation which we will denote as $\mu(s, \theta)$ ...

3

The game of TIC-TAC-TOE can be modelled as a non-deterministic Markov decision process (MDP) if, and only if: The opponent is considered part of the environment. This is a reasonable approach when the goal is to solve playing against a specific opponent. The opponent is using a stochastic policy. Stochastic policies are a generalisation that include ...

3

I think the result you are referring to is the one that says that there always exists a deterministic optimal policy for an MDP. This is true. But note that this does not imply that a stochastic optimal policy can not exist at the same time. Suppose you have an MDP with one state and two actions $a_1$ and $a_2$, both yielding the reward 0 in expectation (as ...

3

You're right! Behaving according to a deterministic policy while still learning would be a terrible idea in most cases (with the exception of environments that "do the exploring for you"; see comments). But deterministic policies are learned off-policy. That is, the experience used to learn the deterministic policy is gathered by behaving according to a ...

2

No it is not possible to use Q-learning to build a deliberately stochastic policy, as the learning algorithm is designed around choosing solely the maximising value at each step, and this assumption carries forward to the action value update step $Q_{k+1}(S_t,A_t) = Q_k(S_t,A_t) + \alpha(R_{t+1} +\gamma\text{max}_{a'}Q_k(S_{t+1},a') - Q_k(S_t,A_t))$ - i.e. ...

1

Is the policy (based in the neural network) a stochastic policy? even if the action space is discrete? Yes. A discrete action space does not require a deterministic policy - it is possible to assign arbitrary probabilities to each action in each state provided each probability is in the range $[0,1]$ and the sum across all allowed actions is $1$. The two ...

Only top voted, non community-wiki answers of a minimum length are eligible