5

Q-learning is guaranteed to converge (in the tabular case) under some mild conditions, one of which is that in the limit we visit each state-action tuple infinitely many times. If your random random policy (i.e. 100% exploration) is guaranteeing this and the other conditions are met (which they probably are) then Q-learning will converge. The reason that ...


3

There are many ideas to escape from local optima in GA. One solution is selecting the population for the next iteration based on the probability that is defined based on the individual score. In that case, you have a chance to select a bad score individual to escape from the local optima. Another efficient solution is playing with the mutation rate to get ...


3

In the discussion about Neil Slater's answer (that he, sadly, deleted) it was pointed out that the policy $\pi$ should also depend on the horizon $h$. The decision of action $a$ can be influenced by how many steps are left. So, the "policy" in that case is actually a collection of policies $\pi_h(a|s)$ indexed by $h$ - the distance to horizon. ...


1

For the case of at least twice differentiable functions, the answer is given by @OmG - you need to look at the eigenvalues of Hessian. For the 1-dimensional case the picture is rather intuitive: If the function grows faster then linear with deviation from the minimum, then the function is convex. For multidimensional case for any projection on a plane, ...


1

It is the same as other functions. You can use Theorem 2 in this lecture (from Princeton University): (ii) condition is about the first-order condition for convexity and (iii) is the second-order. You can also find more detail in chapter 3 of this book ("Convex Optimization" by Stephen Boyd and Lieven Vandenberghe).


1

Even if the mean of the maximum Q-value increases from episode 300 onwards, it doesn't mean that the relative order of the Q-values of the actions that you can take in the states change, which means that the policy may not change, even though the value function changes, assuming you're acting greedily with respect to the value function. More concretely, ...


Only top voted, non community-wiki answers of a minimum length are eligible