2
$\begingroup$

This paper uses image augmentation to improve RL algorithms. It contains the following paragraph - "Our approach, DrQ, is the union of the three separate regularization mechanisms introduced above:

  1. transformations of the input image (Section 3.1).
  2. averaging the Q target over K image transformations (Equation (1)).
  3. averaging the Q function itself over M image transformations (Equation (3))."

I do not understand how part 2 and 3 (Equation 1 and 3) and would highly appreciate some detailed elaboration on it.

Here are the equations -

enter image description here

enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ If possible, you can update with equations mentioned in the paper along with notations for completeness. $\endgroup$
    – hanugm
    Jul 26 '21 at 22:15
2
+50
$\begingroup$

Equation 1

In normal Q-Learning your target is defined as $y_t = r_t + \gamma \mathrm{max_a}Q(s_{t+1}, a)$. Since you're training a regularized version, you construct the estimated value of the next state via averaging your estimations for each image augmentation. To turn this into the expected value over all $k$ transformations for the given state we need to average it by dividing the summed targets by the number of transformations ${k}$.

Equation 3

Here the Q-Function is updated with respect to all the image transformations. $f(s_i, v_{i,m})$ is the transformed image, i.e. it is the same as $s_i$ but its brightness is increased by 0.5. We fit our action value network on the mean squared error between the output of the net and the Q-Target $y_i$ averaged by the number of images transformations and states.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.