1
$\begingroup$

From my understanding, the policy $\pi$ is basically how the agent acts (i.e. the actions it will take in each state).

However, I am confused about the Q value and how it is "affected" by a policy. This answer says

$Q^\pi(s, a)$ is the action-value function. It is the expected return starting from state $s$, following policy $\pi$, taking action $a$. It's focusing on the particular action at the particular state.

From this, I infer that the $Q$ value (the action-value function) will be affected by the policy $\pi$. Why? So, why does the Q value change according to policy $\pi$?

Shouldn't the Q value be constant, because the same action taken in the same state will always give the same yield (and hence remain constantly good/bad)?

All the policy does is find out the max Q values and bases its policy on that information.

$\endgroup$
2
$\begingroup$

First of all, $Q_\pi(s, a)$ IS DEFINED AS the value (i.e. the expected return) of taking some action $a$ in some state $s$, AND THEN following some given policy $\pi$ (until e.g. the end of the game or your life). In other words, suppose that you take action $a$ in state $s$, AND THEN use the policy $\pi$ to behave in the world until you die, then $Q_\pi(s, a)$ would represent the value that you would obtain.

So, we are DEFINING $Q_\pi(s, a)$ in a certain way. This is a DEFINITION! It's not an algorithm. In the algorithms (e.g. Q-learning), things will typically change, but that's a different story that you should investigate later.

From this, I infer that the $Q$ value (the action-value function) will be affected by the policy $\pi$.

So, $Q_\pi(s, a)$ will not keep changing. You could say that $Q_\pi(s, a)$ (which is a function) is "affected by" $\pi$ ONLY in the sense that it is "defined in terms of" $\pi$. To be precise, $Q_\pi(s, a)$ is actually an expectation (which is a mathematical concept similar to an ideal average). If you are not familiar with the concept of expectation, I suggest you get familiar with it first, before studying reinforcement learning.

Shouldn't the Q value be constant, because the same action taken in the same state will always give the same yield (and hence remain constantly good/bad)?

Again, there's the distinction between the algorithm that you actually use to find the function $Q_\pi(s, a)$ and the definition of the same function. In case you are estimating the function with an algorithm, you will not necessarily find "constant Q values". It depends on different aspects, which I would like to avoid discussing here, so that this post doesn't become an open discussion (I suggest you first learn about the basic Bellman equations and then you study the algorithms from the book Reinforcement learning: an introduction by Sutton and Barto).

| improve this answer | |
$\endgroup$
0
$\begingroup$

Ok, Q is the reward associated with being in a given state, following a certain action and then following the given policy.

You need to take the expectation of the sum the immediate reward plus the value function which is defined by the policy.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.