Skip to main content
15 votes
Accepted

What is the difference between a stochastic and a deterministic policy?

A deterministic policy is a function of the form $\pi_{\mathbb{d}}: S \rightarrow A$, that is, a function from the set of states of the environment, $S$, to the set of actions, $A$. The subscript $_{\...
nbro's user avatar
  • 40.9k
11 votes

Is the optimal policy always stochastic if the environment is also stochastic?

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? No. An optimal policy is generally ...
Neil Slater's user avatar
  • 32.7k
7 votes

Did Alphago zero actually beat Alphago 100 games to 0?

Did AlphaGo and AlphaGo [Zero] play 100 repetitions of the same sequence of boards, or were there 100 different games? There were 100 different games. You can view some example games between AlphaGo [...
Neil Slater's user avatar
  • 32.7k
6 votes

Is the optimal policy always stochastic if the environment is also stochastic?

I would say no. For example, consider the multi-armed bandit problem. So, you have $n$ arms which all have a probability of giving you a reward (1 point, for example), $p_i$, $i$ being between 1 and ...
Adrien Forbu's user avatar
4 votes
Accepted

In the policy gradient equation, is $\pi(a_{t} | s_{t}, \theta)$ a distribution or a function?

First, the derivative is usually taken with respect to a variable (input) of the function. Hence the notation $\frac{df}{dx}$ for some function $f(x)$. If you look at your equation more carefully $$\...
nbro's user avatar
  • 40.9k
4 votes

What is the difference between a stochastic and a deterministic policy?

Deterministic Policy : Its means that for every state you have clear defined action you will take For Example: We 100% know we will take action A from state X. Stochastic Policy : Its mean that for ...
Laeeq Khan Niazi's user avatar
3 votes
Accepted

Can a policy with gaussian distribution allow two distinct optimal actions to have distinctively high probabilities?

Assuming by distinct you mean that, for example, the euclidean distance between the two actions is sufficiently large, then no it cannot be true. This is because the Normal distribution is uni-modal. ...
David's user avatar
  • 4,920
2 votes
Accepted

Can Q-learning be used to derive a stochastic policy?

No it is not possible to use Q-learning to build a deliberately stochastic policy, as the learning algorithm is designed around choosing solely the maximising value at each step, and this assumption ...
Neil Slater's user avatar
  • 32.7k
2 votes
Accepted

What's the value of making the RL agent's output stochastic opposed to deterministic?

My understanding of your question is, you have 2 designs: A deterministic policy that outputs 2 scalar for x and y respectively. A value function that outputs the probability of each pixel in the 2D ...
J3soon's user avatar
  • 236
2 votes
Accepted

Is it possible for value-based methods to learn stochastic policies?

Is it possible for value-based methods to learn stochastic policies? Yes, but only in a limited sense, due to the ways it is possible to generate stochastic policies from a value function. For ...
Neil Slater's user avatar
  • 32.7k
2 votes
Accepted

How is $v_*(s) = \max_{\pi} v_\pi(s)$ also applicable in the case of stochastic policies?

The value function is defined as $v_\pi(s) = \mathbb{E}_\pi[G_t | S_t = s]$ where $G_t$ are the (discounted) returns from time step $t$. The expectation is taken with respect to the policy $\pi$ and ...
David's user avatar
  • 4,920
1 vote

Consequence of Dvoretzky Stochastic Approximation Theorem

The Dvoretzky Stochastic Approximation Theorem is a result in the field of stochastic approximation theory, which provides insights into the convergence properties of certain iterative algorithms used ...
mathewgomas's user avatar
1 vote

How do we estimate the value of a stochastic policy?

We are choosing actions randomly with probabilities given by the policy $\pi$. For example, one policy might make two actions A and B equally likely, another might choose A 90% of the time. What we're ...
Lee Reeves's user avatar
1 vote

Is a learned policy, for a deterministic problem, trained in a supervised process, a stochastic policy?

Is the policy (based in the neural network) a stochastic policy? even if the action space is discrete? Yes. A discrete action space does not require a deterministic policy - it is possible to assign ...
Neil Slater's user avatar
  • 32.7k
1 vote

What is the difference between a stochastic and a deterministic policy?

Apart from the answers above, Stochastic Policy function: $\pi (s_1s_2 \dots s_n, a_1 a_2 \dots a_n): \mathcal S \times \mathcal A \rightarrow [0,1]$ is the probability distribution function, that, ...
Abhas Kumar Sinha's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible