0
$\begingroup$

In a Dueling DQN agent (Wang et al.), the Q function is decomposed as

$$ Q(s, a)=V(s) + A(s, a) - \frac{1}{|A|}\sum_{a'\in \mathcal{A}}A(s, a') $$

representing the value of the state, plus the advantage of the action, minus the average advantage of all actions available in that state.

However, this formulation means that the value of the given state is skewed, even after subtracting out the mean advantage of all actions available. Why isn't it set up so that the advantage of the best action is 0 (with the other actions' advantages being negative), thus leading to a more accurate $V(s)$?

$\endgroup$

1 Answer 1

0
$\begingroup$

This is considered in the original paper, but is rejected due to training instability. To quote:

[The average-advantage formulation] increases the stability of the optimization...the advantages only need to change as fast as the mean, instead of having to compensate any change to the optimal action's advantage in [the formulation where $Q(s, a) is normalized with the best action's advantage].

Furthermore, the authors attempt a softmax version, but they find it doesn't perform that differently than the average-advantage version. I couldn't find any additional experiments using this, but it may be worth trying if it is absolutely necessary to obtain an accurate $V(s)$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .