0
$\begingroup$

Q-learning uses the maximizing value at each step, which implies that there is a probability distribution and it happens to choose the one with the highest probability. There is no direct mapping between a particular state to ONLY a particular action but a bunch of actions with varying probabilities. I don't understand.

$\endgroup$
2
  • $\begingroup$ "which implies that there is a probability distribution", what implies that? Probability distribution over what? What does "maximization" have to do with a probability distribution? Q-learning, to compute the target, chooses the value associated with the action of the highest value (i.e. it computes the target as if you assumed that you will take the greedy action). This post is very confusing. Please, next time, before drawing or making possibly wrong assumptions and conclusions, ask a question about what you don't know. That's better than asking a question that is based on wrong assumptions. $\endgroup$ – nbro Nov 30 '20 at 20:50
  • $\begingroup$ @nbro: I think that the OP is honestly confused about what the Q values represent and where probabilities come into it. I am not sure we will get a better description of the issue. $\endgroup$ – Neil Slater Nov 30 '20 at 21:01
2
$\begingroup$

Q-learning uses the maximizing value at each step,

Mostly true. The target policy that Q-learning learns the action values of is the one with the maximum value. While training, Q-learning will take one action randomly (typically with a high probability of taking the action with maximum value), whilst it makes updates to value estimates assuming it will always take the maximising action next.

which implies that there is a probability distribution and it happens to choose the one with the highest probability.

Not true. This is not implied at all. The action values that Q-learning estimates are not probabilities, but expected sums of future reward. The probability distribution for exploring actions is added outside of the Q table or Q function estimate. There is no implied probability distribution of actions in the target policy. When implementing the target policy, you may decide to break ties randomly, but that is not an important detail.

It is worth noting that even if you were working with a table of action probabilities, then an agent that always chose the one with the maximum probability would be deterministic. A list of numbers from a table or non-stochastic function (which is how a Q table or Q function is implemented, and also how policy functions are implemented for methods that do process probabilities), even if it represents probabilities, cannot be random in itself. Instead, it must be interpreted by a process that decides how to use them. A process that includes a random number generator can use the numbers to generate a probability distribution that it samples from.

In Q-learning, the process that updates the Q table estimates does not include a random number generator, so as a consequence it is deterministic. However, the choice of which actions to explore is often random, and it is sometimes the case that the environment includes randomness in state transitions or reward values. So Q-learning taken as a whole is a random process.

$\endgroup$
2
  • $\begingroup$ Based on the last paragraph, is it sufficient to conclude whether or not a policy is stochastic or deterministic depends on how the TD target ( max Q for Q learning) is updated, and not depends on behavior policy? $\endgroup$ – Aung Khant Nov 30 '20 at 16:36
  • $\begingroup$ @AungKhant You should read and understand any RL method you want to understand in detail, as the functions used vary a lot. As a rule of thumb most RL control methods will converge towards deterministic policies, but many will represent a current target policy which is stochastic. Q learning and other off-policy value-based methods have target policies which are not random, but are instead deterministic greedy action choice over the current value estimates. $\endgroup$ – Neil Slater Nov 30 '20 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.