Right, my exploration function was meant as 'upgrade' from a strictly e-greedy strategy (to mitigate thrashing by the time the optimal policy is learned). But I don't get why then it won't work even if I only use it in the action selection (behavior policy). Also the idea of plugging it in the update step I think is to propagate the optimism about exploring not only to the state in question but to states leading to that state. As for separating behavior and learning, shouldn't the behavior be driven from what has been learned so far?