Am I right to say that the Q value of a particular state and action is the same as the state-action pair value of that same state and action?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
I don't understand your question very clearly.
Q-value of a particular state-action pair (s,a) under policy $\pi$ is the total reward you would expect to collect if you start from the state s, take the action a, and follow policy $\pi$ from then on.
In the literature, this is referred to as state-action values.
-
$\begingroup$ So, the Q value is not a value assigned to individual states and their actions? Instead, the q value is the sum of rewards I would expect to get , starting from the current state to the terminal state, following policy π? Also, the value that I would assign to the individual states and their actions to measure how good that action is, is called the state-action value? $\endgroup$– BG10Apr 16, 2020 at 0:59
-
$\begingroup$ I think you are getting a bit confused. State value functions (represented by V(s)) is the total sum of rewards you would expect to get if you start at state s, and go to the terminal state. On the other hand, state-action value functions, denoted by Q(s,a) is a function of both the state and action. It denotes the total expected reward that you'll get it you start in state s and take action a in state s. According to this, you have to take action a in state s. It doesn't matter how you act in the remaining states. Does this help? $\endgroup$– peceyApr 16, 2020 at 4:02
-
$\begingroup$ So for the state value functions and state-action value functions, all they tell you is how good it is for an agent to be in that particular state(and perform a particular action for the Q(s,a) ) while following the policy π . We will always be changing states during an episode ( every action we take means changing to another state) and will get to choose the state we will be in next. Using the state value function, we will be able to know which is the best possible state ( the one with the highest state value function) and will go to that state because we know it's the best for our policy π $\endgroup$– BG10Apr 16, 2020 at 6:10
$\begingroup$
$\endgroup$
3
Am I right to say that the Q value of a particular state and action is the same as the state-action pair value of that same state and action?
In general $Q$ is used as the symbol for action value and $Q(s,a)$ is the action value function.
The phrase "state-action pair value" is used to mean the same thing in some texts.
So, yes, you are right. At least I cannot think of or find any counter-examples where the two things could be used to refer to different things in the same text.
It is possible to work with different formulae for Q - e.g. finite horizon, discounted, average reward. There are also conceptual differences between a "true" action value, what you are currently estimating, and implementation details such as how you implement a Q function in code. In addition, there is also the advantage function usually labelled $A(s,a)$ which might also be considered as a "state-action pair value" by some authors.
However, those would not generally be flagged in documents by labelling one a "Q value" and another "state-action pair value" without at least some other text. So in general you are safe to consider the two terms to mean the same thing.
answered Apr 15, 2020 at 14:28
-
$\begingroup$ Yes, I feel the action value is a little confusing to me in the sense that it can mean the 'true' or 'estimated' value. However, isn't the action value based on the rewards, which is constant all the time?(because action value states how good it is to perform a certain action a in state s) If so, then shouldn't the action value be constant also? Also, we 'estimate' the action value because we are lacking the transition and reward probabilities? Thanks! $\endgroup$– BG10Apr 16, 2020 at 0:42
-
$\begingroup$ @BG10 Yes, right, it's assumed that there's an optimal value function (either state-action or just state value functions) and then there's an estimate of the optimal value function, which is what you have or want to find. While estimating this value function, e.g. with Q-learning, you will not necessarily encounter "constant rewards all the time". For example, in the case of Q-learning, while exploring the environment, you will have to take actions to "explore" (unknown parts of the environment) and to "exploit" (get more reward from actions that you currently think are the best). $\endgroup$– nbroApr 16, 2020 at 1:01
-
$\begingroup$ @BG10 Typically, you will trade-off between exploration and exploitation, and this often done randomly, e.g. with a the $\epsilon$-greedy policy. So, no, in general, you will not encounter "constant rewards" even if the environment is deterministic (i.e. the transition probabilities and reward function do not change). If you need more clarifications, I think you should ask a new question on the site. Actually, you should ask this question on the site anyway! $\endgroup$– nbroApr 16, 2020 at 1:02