12 votes
Accepted

What is the difference between a stochastic and a deterministic policy?

A deterministic policy is a function of the form $\pi_{\mathbb{d}}: S \rightarrow A$, that is, a function from the set of states of the environment, $S$, to the set of actions, $A$. The subscript $_{\...
user avatar
  • 33.8k
12 votes
Accepted

Why is the derivative of this objective function 0 if the policy is deterministic?

Here is the gradient that they are discussing in the video: $$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \...
user avatar
  • 9,379
9 votes
Accepted

What does "stationary" mean in the context of reinforcement learning?

A stationary policy is a policy that does not change. Although strictly that is a time-dependent issue, that is not what the distinction refers to in reinforcement learning. It generally means that ...
user avatar
  • 23.8k
9 votes

Is the optimal policy always stochastic if the environment is also stochastic?

Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic? No. An optimal policy is generally ...
user avatar
  • 23.8k
7 votes

Why is the policy not a part of the MDP definition?

The MDP defines the environment (which corresponds to the task that you need to solve), so it defines e.g. the states of the environment, the actions that you can take in those states, the ...
user avatar
  • 33.8k
7 votes
Accepted

Is Q-learning only capable of learning a deterministic policy?

If we assume a tabular setting, then Q-learning converges to the optimal state-action value function, from which an optimal policy can be derived, provided a few conditions are met. In finite MDPs, ...
user avatar
  • 33.8k
6 votes

Is the optimal policy always stochastic if the environment is also stochastic?

I would say no. For example, consider the multi-armed bandit problem. So, you have $n$ arms which all have a probability of giving you a reward (1 point, for example), $p_i$, $i$ being between 1 and ...
user avatar
5 votes
Accepted

What is the difference between a stationary and a non-stationary policy?

A stationary policy, $\pi_t$, is a policy that does not change over time, that is, $\pi_t = \pi, \forall t \geq 0$, where $\pi$ can either be a function, $\pi: S \rightarrow A$ (a deterministic policy)...
user avatar
  • 33.8k
5 votes
Accepted

Are there some notions of distance between two policies?

Given that policies are probability distributions, in principle, you can use any metric or measure of distance that can be used to compare two probability distributions. (Note that notions of distance ...
user avatar
  • 33.8k
4 votes
Accepted

What is the relation between a policy which is the solution to a MDP and a policy like $\epsilon$-greedy?

for example, the "greedy policy" always chooses the action with the highest expected return, no matter which state we are in The "no matter which state we are in" there is generally not true; in ...
user avatar
  • 9,379
4 votes
Accepted

Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?

You appear to comparing the value table update steps in policy iteration and value iteration, which are both derived from Bellman equations. Policy iteration In policy iteration, a policy lookup table ...
user avatar
  • 23.8k
4 votes
Accepted

Are optimal policies always deterministic, or can there also be optimal policies that are stochastic?

I think the result you are referring to is the one that says that there always exists a deterministic optimal policy for an MDP. This is true. But note that this does not imply that a stochastic ...
user avatar
  • 381
4 votes
Accepted

In reinforcement learning, why are policies defined as functions of states and not observations?

Ultimately, a policy must be such that is is possible for an agent to execute it. If the policy depends on the state, the implicit assumption is that the agent has knowledge of the state and can ...
user avatar
  • 381
3 votes
Accepted

A few questions regarding the difference between policy iteration and value iteration

$\pi(s)$ does not mean $q(s,a)$ here. $\pi(s)$ is a policy that represents probability distribution over action space for a specific state. $q(s,a)$ is a state-action pair value function that tells us ...
user avatar
  • 2,226
3 votes

Why is the derivative of this objective function 0 if the policy is deterministic?

Well, I'd rather comment, but I don't have yet this privilege, so here are some comments. First, having a deterministic policy inside the log would do create trivial terms. Secondly, for me, in ...
user avatar
  • 571
3 votes

What does "stationary" mean in the context of reinforcement learning?

A stationary policy is the one that does not depend on time. Meaning that the agent will take the same decision whenever certain conditions are met. This stationary policy may be probabilistic which ...
user avatar
3 votes
Accepted

Why is there an inconsistency between my calculations of Policy Iteration and this Sutton & Barto's diagram?

Your calculations are correct, but you have misinterpreted the equations and the diagram. The index $k$ in $v_k$ for the diagram refers to the policy evaluation update iteration only, and is not ...
user avatar
  • 23.8k
3 votes
Accepted

What is meant by "generate the data" in describing the difference between on-policy and off-policy?

In the book, the phrase "generate the data" refers to the data from observations about states, actions, next states and rewards, that then get used to make value estimate updates. In both ...
user avatar
  • 23.8k
3 votes
Accepted

Why is the optimal policy for an infinite horizon MDP deterministic?

Suppose you learned your action-value function perfectly. Recall that the action-value function measures the expected return after taking a given action in a given state. Now, the goal when solving an ...
user avatar
  • 961
3 votes
Accepted

What is a "learned policy" in Q-learning?

A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed ...
user avatar
  • 23.8k
3 votes
Accepted

How can we find the value function by solving a system of linear equations without knowing the policy?

Your equations all look correct to me. It is not possible to solve the linear equation for state values in the vector $V$ without knowing the policy. There are ways of working with MDPs, through ...
user avatar
  • 23.8k
2 votes

What does "stationary" mean in the context of reinforcement learning?

You are right: a stationary policy is independent of time. It is basically a mapping from states to actions (or probability distributions over actions). Regardless of the point in time in which the ...
user avatar
  • 81
2 votes
Accepted

An example of a unique value function which is associated with multiple optimal policies

Consider a very simple grid-world, consisting of 4 cells, where an agent starts in the bottom-left corner, has actions to move North/East/South/West, and receives a reward $R = 1$ for reaching the top-...
user avatar
  • 9,379
2 votes
Accepted

Why does the value of state change depending on the policy used to get to that state?

I guess I'm having difficulty grasping the concept that the goodness of a state changes depending on how an agent got there It doesn't. The value of a state changes depending on what the agent will ...
user avatar
  • 23.8k
2 votes
Accepted

Possible inconsistency in the Policy Improvement equation

Why are they comparing state value function to action value function? It is because $v_{\pi}(s)$ and $q_{\pi}(s,a)$ measure the same quantity at different stages of the trajectory. By comparing the ...
user avatar
  • 23.8k
2 votes

What is the difference between a stochastic and a deterministic policy?

Deterministic Policy : Its means that for every state you have clear defined action you will take For Example: We 100% know we will take action A from state X. Stochastic Policy : Its mean that for ...
user avatar
2 votes

What is the difference between a non-stationary policy and a state that stores time?

So, here's is the question: Is it true that a non-stationary policy must satisfy this condition? $$ \forall i, j \in \mathbb{N}, s \in S, \pi (i, s) = \pi(j, s) $$ With your custom notation (...
user avatar
  • 9,379
2 votes

Is the policy really invariant under affine transformations of the reward function?

This statement: (it is well-known that the optimal policy is invariant to positive affine transformation of the reward function). is, as far as I know, and as you summarise, incorrect* because ...
user avatar
  • 23.8k
2 votes
Accepted

Can someone please help me validate my MDP?

The good news is that: Your MDP appears valid, with well-defined states, actions. It has state transition and reward functions (which you have implemented as matrices). There is nothing else to add, ...
user avatar
  • 23.8k

Only top scored, non community-wiki answers of a minimum length are eligible