Skip to main content
15 votes
Accepted

What is the difference between a stochastic and a deterministic policy?

A deterministic policy is a function of the form $\pi_{\mathbb{d}}: S \rightarrow A$, that is, a function from the set of states of the environment, $S$, to the set of actions, $A$. The subscript $_{\...
nbro's user avatar
  • 40.9k
13 votes
Accepted

Why is the derivative of this objective function 0 if the policy is deterministic?

Here is the gradient that they are discussing in the video: $$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \...
Dennis Soemers's user avatar
  • 10.3k
11 votes

Is the optimal policy always stochastic if the environment is also stochastic?

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? No. An optimal policy is generally ...
Neil Slater's user avatar
  • 32.6k
9 votes
Accepted

What does "stationary" mean in the context of reinforcement learning?

A stationary policy is a policy that does not change. Although strictly that is a time-dependent issue, that is not what the distinction refers to in reinforcement learning. It generally means that ...
Neil Slater's user avatar
  • 32.6k
8 votes
Accepted

Is Q-learning only capable of learning a deterministic policy?

If we assume a tabular setting, then Q-learning converges to the optimal state-action value function, from which an optimal policy can be derived, provided a few conditions are met. In finite MDPs, ...
nbro's user avatar
  • 40.9k
7 votes

Why is the policy not a part of the MDP definition?

The MDP defines the environment (which corresponds to the task that you need to solve), so it defines e.g. the states of the environment, the actions that you can take in those states, the ...
nbro's user avatar
  • 40.9k
6 votes
Accepted

What is the difference between a stationary and a non-stationary policy?

A stationary policy, $\pi_t$, is a policy that does not change over time, that is, $\pi_t = \pi, \forall t \geq 0$, where $\pi$ can either be a function, $\pi: S \rightarrow A$ (a deterministic policy)...
nbro's user avatar
  • 40.9k
6 votes

Is the optimal policy always stochastic if the environment is also stochastic?

I would say no. For example, consider the multi-armed bandit problem. So, you have $n$ arms which all have a probability of giving you a reward (1 point, for example), $p_i$, $i$ being between 1 and ...
Adrien Forbu's user avatar
5 votes
Accepted

What is the relation between a policy which is the solution to a MDP and a policy like $\epsilon$-greedy?

for example, the "greedy policy" always chooses the action with the highest expected return, no matter which state we are in The "no matter which state we are in" there is generally not true; in ...
Dennis Soemers's user avatar
  • 10.3k
5 votes
Accepted

Are there some notions of distance between two policies?

Given that policies are probability distributions, in principle, you can use any metric or measure of distance that can be used to compare two probability distributions. (Note that notions of distance ...
nbro's user avatar
  • 40.9k
5 votes
Accepted

Are optimal policies always deterministic, or can there also be optimal policies that are stochastic?

I think the result you are referring to is the one that says that there always exists a deterministic optimal policy for an MDP. This is true. But note that this does not imply that a stochastic ...
mikkola's user avatar
  • 579
4 votes

What is the difference between a stochastic and a deterministic policy?

Deterministic Policy : Its means that for every state you have clear defined action you will take For Example: We 100% know we will take action A from state X. Stochastic Policy : Its mean that for ...
Laeeq Khan Niazi's user avatar
4 votes
Accepted

Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?

You appear to comparing the value table update steps in policy iteration and value iteration, which are both derived from Bellman equations. Policy iteration In policy iteration, a policy lookup table ...
Neil Slater's user avatar
  • 32.6k
4 votes
Accepted

What is a "learned policy" in Q-learning?

A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed ...
Neil Slater's user avatar
  • 32.6k
4 votes
Accepted

In reinforcement learning, why are policies defined as functions of states and not observations?

Ultimately, a policy must be such that is is possible for an agent to execute it. If the policy depends on the state, the implicit assumption is that the agent has knowledge of the state and can ...
mikkola's user avatar
  • 579
3 votes
Accepted

If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy?

The output layer of the network contains one unit, telling me the Q value of the provided state with the assumption that the action taken in that state will be determined by the policy. Typically in ...
Neil Slater's user avatar
  • 32.6k
3 votes
Accepted

A few questions regarding the difference between policy iteration and value iteration

$\pi(s)$ does not mean $q(s,a)$ here. $\pi(s)$ is a policy that represents probability distribution over action space for a specific state. $q(s,a)$ is a state-action pair value function that tells us ...
Brale's user avatar
  • 2,406
3 votes

Why is the derivative of this objective function 0 if the policy is deterministic?

Well, I'd rather comment, but I don't have yet this privilege, so here are some comments. First, having a deterministic policy inside the log would do create trivial terms. Secondly, for me, in ...
16Aghnar's user avatar
  • 601
3 votes

What does "stationary" mean in the context of reinforcement learning?

A stationary policy is the one that does not depend on time. Meaning that the agent will take the same decision whenever certain conditions are met. This stationary policy may be probabilistic which ...
Khalid Ibrahim's user avatar
3 votes
Accepted

Why is there an inconsistency between my calculations of Policy Iteration and this Sutton & Barto's diagram?

Your calculations are correct, but you have misinterpreted the equations and the diagram. The index $k$ in $v_k$ for the diagram refers to the policy evaluation update iteration only, and is not ...
Neil Slater's user avatar
  • 32.6k
3 votes
Accepted

What is meant by "generate the data" in describing the difference between on-policy and off-policy?

In the book, the phrase "generate the data" refers to the data from observations about states, actions, next states and rewards, that then get used to make value estimate updates. In both ...
Neil Slater's user avatar
  • 32.6k
3 votes
Accepted

Why is the optimal policy for an infinite horizon MDP deterministic?

Suppose you learned your action-value function perfectly. Recall that the action-value function measures the expected return after taking a given action in a given state. Now, the goal when solving an ...
harwiltz's user avatar
  • 1,136
3 votes
Accepted

How can we find the value function by solving a system of linear equations without knowing the policy?

Your equations all look correct to me. It is not possible to solve the linear equation for state values in the vector $V$ without knowing the policy. There are ways of working with MDPs, through ...
Neil Slater's user avatar
  • 32.6k
3 votes

If a policy is epsilon-greedy, is it technically stochastic?

I would argue it is just stochastic because it chooses the current best action with probability $1-\epsilon+\epsilon/|A|$ and then selects randomly among the rest of the actions with the remaining ...
sma's user avatar
  • 823
2 votes

What does "stationary" mean in the context of reinforcement learning?

You are right: a stationary policy is independent of time. It is basically a mapping from states to actions (or probability distributions over actions). Regardless of the point in time in which the ...
D. S.'s user avatar
  • 81
2 votes
Accepted

An example of a unique value function which is associated with multiple optimal policies

Consider a very simple grid-world, consisting of 4 cells, where an agent starts in the bottom-left corner, has actions to move North/East/South/West, and receives a reward $R = 1$ for reaching the top-...
Dennis Soemers's user avatar
  • 10.3k
2 votes
Accepted

Why does the value of state change depending on the policy used to get to that state?

I guess I'm having difficulty grasping the concept that the goodness of a state changes depending on how an agent got there It doesn't. The value of a state changes depending on what the agent will ...
Neil Slater's user avatar
  • 32.6k
2 votes
Accepted

Possible inconsistency in the Policy Improvement equation

Why are they comparing state value function to action value function? It is because $v_{\pi}(s)$ and $q_{\pi}(s,a)$ measure the same quantity at different stages of the trajectory. By comparing the ...
Neil Slater's user avatar
  • 32.6k
2 votes

What is the difference between a non-stationary policy and a state that stores time?

So, here's is the question: Is it true that a non-stationary policy must satisfy this condition? $$ \forall i, j \in \mathbb{N}, s \in S, \pi (i, s) = \pi(j, s) $$ With your custom notation (...
Dennis Soemers's user avatar
  • 10.3k

Only top scored, non community-wiki answers of a minimum length are eligible