Tag Info

9

A deterministic policy is a function of the form $\pi_{\mathbb{d}}: S \rightarrow A$, that is, a function from the set of states of the environment, $S$, to the set of actions, $A$. The subscript $_{\mathbb{d}}$ only indicates that this is a ${\mathbb{d}}$eterministic policy. For example, in a grid world, the set of states of the environment, $S$, is ...

8

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? No. An optimal policy is generally deterministic unless: Important state information is missing (a POMDP). For example, in a map where the agent is not allowed to know its exact location or remember ...

6

Did AlphaGo and AlphaGo [Zero] play 100 repetitions of the same sequence of boards, or were there 100 different games? There were 100 different games. You can view some example games between AlphaGo [Lee] and AlphaGo Zero here. They are clearly all different. This statement in the question shows a misunderstanding: My understanding of AlphaGo and AlphaGo [...

5

I would say no. For example, consider the multi-armed bandit problem. So, you have $n$ arms which all have a probability of giving you a reward (1 point, for example), $p_i$, $i$ being between 1 and $n$. This is a simple stochastic environment: this is a one state environment, but it is still an environment. But obviously the optimal policy is to choose ...

3

This update rule can still be applied in the continuous domain. As pointed out in the comments, suppose we are parameterising our policy using a Gaussian distribution, where our neural networks take as input the state we are in and output the parameters of a Gaussian distribution, the mean and the standard deviation which we will denote as $\mu(s, \theta)$ ...

3

You're right! Behaving according to a deterministic policy while still learning would be a terrible idea in most cases (with the exception of environments that "do the exploring for you"; see comments). But deterministic policies are learned off-policy. That is, the experience used to learn the deterministic policy is gathered by behaving according to a ...

2

No it is not possible to use Q-learning to build a deliberately stochastic policy, as the learning algorithm is designed around choosing solely the maximising value at each step, and this assumption carries forward to the action value update step $Q_{k+1}(S_t,A_t) = Q_k(S_t,A_t) + \alpha(R_{t+1} +\gamma\text{max}_{a'}Q_k(S_{t+1},a') - Q_k(S_t,A_t))$ - i.e. ...

1

Is the policy (based in the neural network) a stochastic policy? even if the action space is discrete? Yes. A discrete action space does not require a deterministic policy - it is possible to assign arbitrary probabilities to each action in each state provided each probability is in the range $[0,1]$ and the sum across all allowed actions is $1$. The two ...

1

Deterministic Policy : Its means that for every state you have clear defined action you will take For Example: We 100% know we will take action A from state X. Stochastic Policy : Its mean that for every state you do not have clear defined action to take but you have probability distribution for actions to take from that state. For example there are 10% ...

Only top voted, non community-wiki answers of a minimum length are eligible