What do equations 1 and 3 describe in the "Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels" paper?

Question

This paper uses image augmentation to improve RL algorithms. It contains the following paragraph - "Our approach, DrQ, is the union of the three separate regularization mechanisms introduced above:

1. transformations of the input image (Section 3.1).
2. averaging the Q target over K image transformations (Equation (1)).
3. averaging the Q function itself over M image transformations (Equation (3))."

I do not understand how part 2 and 3 (Equation 1 and 3) and would highly appreciate some detailed elaboration on it.

Here are the equations -

Answer 1

Equation 1

In normal Q-Learning your target is defined as $y_t = r_t + \gamma \mathrm{max_a}Q(s_{t+1}, a)$. Since you're training a regularized version, you construct the estimated value of the next state via averaging your estimations for each image augmentation. To turn this into the expected value over all $k$ transformations for the given state we need to average it by dividing the summed targets by the number of transformations ${k}$.

Equation 3

Here the Q-Function is updated with respect to all the image transformations. $f(s_i, v_{i,m})$ is the transformed image, i.e. it is the same as $s_i$ but its brightness is increased by 0.5. We fit our action value network on the mean squared error between the output of the net and the Q-Target $y_i$ averaged by the number of images transformations and states.