0
$\begingroup$

In a Dueling DQN agent (Wang et al.), the Q function is decomposed as

$$ Q(s, a)=V(s) + A(s, a) - \frac{1}{|A|}\sum_{a'\in \mathcal{A}}A(s, a') $$

representing the value of the state, plus the advantage of the action, minus the average advantage of all actions available in that state.

However, this formulation means that the value of the given state is skewed, even after subtracting out the mean advantage of all actions available. Why isn't it set up so that the advantage of the best action is 0 (with the other actions' advantages being negative), thus leading to a more accurate $V(s)$?

$\endgroup$
0
$\begingroup$

This is considered in the original paper, but is rejected due to training instability. To quote:

[The average-advantage formulation] increases the stability of the optimization...the advantages only need to change as fast as the mean, instead of having to compensate any change to the optimal action's advantage in [the formulation where $Q(s, a) is normalized with the best action's advantage].

Furthermore, the authors attempt a softmax version, but they find it doesn't perform that differently than the average-advantage version. I couldn't find any additional experiments using this, but it may be worth trying if it is absolutely necessary to obtain an accurate $V(s)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.