15
votes
Accepted
What is the difference between a stochastic and a deterministic policy?
A deterministic policy is a function of the form $\pi_{\mathbb{d}}: S \rightarrow A$, that is, a function from the set of states of the environment, $S$, to the set of actions, $A$. The subscript $_{\...
- 39.1k
13
votes
Accepted
Why is the derivative of this objective function 0 if the policy is deterministic?
Here is the gradient that they are discussing in the video:
$$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \...
- 9,944
11
votes
Is the optimal policy always stochastic if the environment is also stochastic?
Is the optimal policy always stochastic (that is, a map from states to a probability distribution over actions) if the environment is also stochastic?
No.
An optimal policy is generally ...
- 29k
9
votes
Accepted
What does "stationary" mean in the context of reinforcement learning?
A stationary policy is a policy that does not change. Although strictly that is a time-dependent issue, that is not what the distinction refers to in reinforcement learning. It generally means that ...
- 29k
8
votes
Accepted
Is Q-learning only capable of learning a deterministic policy?
If we assume a tabular setting, then Q-learning converges to the optimal state-action value function, from which an optimal policy can be derived, provided a few conditions are met.
In finite MDPs, ...
- 39.1k
7
votes
Why is the policy not a part of the MDP definition?
The MDP defines the environment (which corresponds to the task that you need to solve), so it defines e.g. the states of the environment, the actions that you can take in those states, the ...
- 39.1k
6
votes
Accepted
What is the difference between a stationary and a non-stationary policy?
A stationary policy, $\pi_t$, is a policy that does not change over time, that is, $\pi_t = \pi, \forall t \geq 0$, where $\pi$ can either be a function, $\pi: S \rightarrow A$ (a deterministic policy)...
- 39.1k
6
votes
Is the optimal policy always stochastic if the environment is also stochastic?
I would say no.
For example, consider the multi-armed bandit problem. So, you have $n$ arms which all have a probability of giving you a reward (1 point, for example), $p_i$, $i$ being between 1 and ...
- 251
5
votes
Accepted
Are there some notions of distance between two policies?
Given that policies are probability distributions, in principle, you can use any metric or measure of distance that can be used to compare two probability distributions. (Note that notions of distance ...
- 39.1k
5
votes
Accepted
Are optimal policies always deterministic, or can there also be optimal policies that are stochastic?
I think the result you are referring to is the one that says that there always exists a deterministic optimal policy for an MDP. This is true. But note that this does not imply that a stochastic ...
- 580
4
votes
Accepted
What is the relation between a policy which is the solution to a MDP and a policy like $\epsilon$-greedy?
for example, the "greedy policy" always chooses the action with the highest expected return, no matter which state we are in
The "no matter which state we are in" there is generally not true; in ...
- 9,944
4
votes
Accepted
Why doesn't value iteration use $\pi(a \mid s)$ while policy evaluation does?
You appear to comparing the value table update steps in policy iteration and value iteration, which are both derived from Bellman equations.
Policy iteration
In policy iteration, a policy lookup table ...
- 29k
4
votes
Accepted
What is a "learned policy" in Q-learning?
A Q table allows you to look up any state/action pair in it and find the associated action value. It is not itself a policy. However, in order to calculate the action values, you will have assumed ...
- 29k
4
votes
Accepted
In reinforcement learning, why are policies defined as functions of states and not observations?
Ultimately, a policy must be such that is is possible for an agent to execute it.
If the policy depends on the state, the implicit assumption is that the agent has knowledge of the state and can ...
- 580
3
votes
Accepted
If deep Q learning involves adjusting the value function for a specific policy, then how do I choose the right policy?
The output layer of the network contains one unit, telling me the Q value of the provided state with the assumption that the action taken in that state will be determined by the policy.
Typically in ...
- 29k
3
votes
Accepted
A few questions regarding the difference between policy iteration and value iteration
$\pi(s)$ does not mean $q(s,a)$ here. $\pi(s)$ is a policy that represents probability distribution over action space for a specific state. $q(s,a)$ is a state-action pair value function that tells us ...
- 2,316
3
votes
Why is the derivative of this objective function 0 if the policy is deterministic?
Well, I'd rather comment, but I don't have yet this privilege, so here are some comments.
First, having a deterministic policy inside the log would do create trivial terms.
Secondly, for me, in ...
- 591
3
votes
What does "stationary" mean in the context of reinforcement learning?
A stationary policy is the one that does not depend on time. Meaning that the agent will take the same decision whenever certain conditions are met. This stationary policy may be probabilistic which ...
3
votes
Accepted
Why is there an inconsistency between my calculations of Policy Iteration and this Sutton & Barto's diagram?
Your calculations are correct, but you have misinterpreted the equations and the diagram. The index $k$ in $v_k$ for the diagram refers to the policy evaluation update iteration only, and is not ...
- 29k
3
votes
Accepted
What is meant by "generate the data" in describing the difference between on-policy and off-policy?
In the book, the phrase "generate the data" refers to the data from observations about states, actions, next states and rewards, that then get used to make value estimate updates.
In both ...
- 29k
3
votes
Accepted
Why is the optimal policy for an infinite horizon MDP deterministic?
Suppose you learned your action-value function perfectly. Recall that the action-value function measures the expected return after taking a given action in a given state. Now, the goal when solving an ...
- 1,106
3
votes
Accepted
How can we find the value function by solving a system of linear equations without knowing the policy?
Your equations all look correct to me.
It is not possible to solve the linear equation for state values in the vector $V$ without knowing the policy.
There are ways of working with MDPs, through ...
- 29k
3
votes
If a policy is epsilon-greedy, is it technically stochastic?
I would argue it is just stochastic because it chooses the current best action with probability $1-\epsilon+\epsilon/|A|$ and then selects randomly among the rest of the actions with the remaining ...
- 823
2
votes
What does "stationary" mean in the context of reinforcement learning?
You are right: a stationary policy is independent of time. It is basically a mapping from states to actions (or probability distributions over actions). Regardless of the point in time in which the ...
- 81
2
votes
Accepted
An example of a unique value function which is associated with multiple optimal policies
Consider a very simple grid-world, consisting of 4 cells, where an agent starts in the bottom-left corner, has actions to move North/East/South/West, and receives a reward $R = 1$ for reaching the top-...
- 9,944
2
votes
Accepted
Why does the value of state change depending on the policy used to get to that state?
I guess I'm having difficulty grasping the concept that the goodness of a state changes depending on how an agent got there
It doesn't.
The value of a state changes depending on what the agent will ...
- 29k
2
votes
Accepted
Possible inconsistency in the Policy Improvement equation
Why are they comparing state value function to action value function?
It is because $v_{\pi}(s)$ and $q_{\pi}(s,a)$ measure the same quantity at different stages of the trajectory. By comparing the ...
- 29k
2
votes
What is the difference between a stochastic and a deterministic policy?
Deterministic Policy :
Its means that for every state you have clear defined action you will take
For Example: We 100% know we will take action A from state X.
Stochastic Policy :
Its mean that for ...
- 121
2
votes
What is the difference between a non-stationary policy and a state that stores time?
So, here's is the question: Is it true that a non-stationary policy must satisfy this condition?
$$ \forall i, j \in \mathbb{N}, s \in S, \pi (i, s) = \pi(j, s) $$
With your custom notation (...
- 9,944
Only top scored, non community-wiki answers of a minimum length are eligible