# What do equations 1 and 3 describe in the "Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels" paper?

This paper uses image augmentation to improve RL algorithms. It contains the following paragraph - "Our approach, DrQ, is the union of the three separate regularization mechanisms introduced above:

1. transformations of the input image (Section 3.1).
2. averaging the Q target over K image transformations (Equation (1)).
3. averaging the Q function itself over M image transformations (Equation (3))."

I do not understand how part 2 and 3 (Equation 1 and 3) and would highly appreciate some detailed elaboration on it.

Here are the equations -

• If possible, you can update with equations mentioned in the paper along with notations for completeness. Jul 26 '21 at 22:15

In normal Q-Learning your target is defined as $$y_t = r_t + \gamma \mathrm{max_a}Q(s_{t+1}, a)$$. Since you're training a regularized version, you construct the estimated value of the next state via averaging your estimations for each image augmentation. To turn this into the expected value over all $$k$$ transformations for the given state we need to average it by dividing the summed targets by the number of transformations $${k}$$.
Here the Q-Function is updated with respect to all the image transformations. $$f(s_i, v_{i,m})$$ is the transformed image, i.e. it is the same as $$s_i$$ but its brightness is increased by 0.5. We fit our action value network on the mean squared error between the output of the net and the Q-Target $$y_i$$ averaged by the number of images transformations and states.