2
$\begingroup$

This paper uses image augmentation to improve RL algorithms. It contains the following paragraph - "Our approach, DrQ, is the union of the three separate regularization mechanisms introduced above:

  1. transformations of the input image (Section 3.1).
  2. averaging the Q target over K image transformations (Equation (1)).
  3. averaging the Q function itself over M image transformations (Equation (3))."

I do not understand how part 2 and 3 (Equation 1 and 3) and would highly appreciate some detailed elaboration on it.

Here are the equations -

enter image description here

enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ If possible, you can update with equations mentioned in the paper along with notations for completeness. $\endgroup$
    – hanugm
    Jul 26, 2021 at 22:15

1 Answer 1

2
+50
$\begingroup$

Equation 1

In normal Q-Learning your target is defined as $y_t = r_t + \gamma \mathrm{max_a}Q(s_{t+1}, a)$. Since you're training a regularized version, you construct the estimated value of the next state via averaging your estimations for each image augmentation. To turn this into the expected value over all $k$ transformations for the given state we need to average it by dividing the summed targets by the number of transformations ${k}$.

Equation 3

Here the Q-Function is updated with respect to all the image transformations. $f(s_i, v_{i,m})$ is the transformed image, i.e. it is the same as $s_i$ but its brightness is increased by 0.5. We fit our action value network on the mean squared error between the output of the net and the Q-Target $y_i$ averaged by the number of images transformations and states.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .