0
$\begingroup$

Is the following sentence correct?

The estimated values $Q(a)$ do not converge to the true values $q_*(a)$ because $\epsilon$-greedy action selection behaves randomly from time to time.

My Answer: The sentence is wrong since eventually the estimated value $Q(a)$ will converge to $q_*(a)$ and the randomness of epsilon greedy action selection won't be able to influence it if we run it for a long time. Is my answer correct?

Improve this question
$\endgroup$
2
  • $\begingroup$ @Faizy Thanks but could you please explain why? I don't get why randomness is a problem for the convergence of estimated Q to the true value q. $\endgroup$
    – Gunners
    Oct 18, 2022 at 15:23
  • $\begingroup$ i think you correct.... Running the epsilon greedy algorithm for a long time will minimize the influence of randomness on the estimated value $Q(a)$. The epsilon greedy action randomly selects actions with probability epsilon and otherwise acts greedily with respect to the current estimated value of $Q(a)$. Over time, the algorithm will tend to converge on the true values of $q_∗(a)$, since the random actions will be less influential. $\endgroup$
    – Faizy
    Oct 18, 2022 at 18:22

1 Answer 1

Reset to default
1
$\begingroup$

I believe, that you are correct in saying the sentence is incorrect. Since, as you mention, the epsilon greedy action selection will just allow for the Q values to converge to the optimal q* value since the actions of the policy just provide more observations for Q which will allow for the q* to be converged to. However, this is assuming that the updating of the Q values is in such a way that each new observation has less of an effect on the Q(a). Though it is not the case that the epsilon greedy will converge to the optimal policy/action due to the randomness mentioned by the answer.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .