How to convert sequences of images into state in DQN?

Question

I recently read the DQN paper titled: Playing Atari with Deep Reinforcement Learning. My basic and rough understanding of the paper is as follows:

You have two neural networks; one stays frozen for a duration of time steps and is used in the computation of the loss function with the neural network that is updating. The loss function is used to update the neural network using gradient descent.

Experience replay is used, which basically creates a buffer of experiences. This buffer of experiences is randomly sampled and these random samples are used to update the non-frozen neural network.

My question pertains to the DQN algorithm illustrated in the paper: Algorithm 1, more specifically lines 4 and 9 of this algorithm. My understanding, which is also mentioned early on in the paper, is that the states are actually sequences of the game-play frames. I want to know, since the input is given to a CNN, how would we encode these frames to serve as input to the CNN?

I also want to know since $s_{1}$ is equal to a set, which can be seen in line 4 of the algorithm, then why is $s_{t+1}$ equal to $s_{t}$, $a_{t}$, $x_{t+1}$?

Answer 1

I want to know, since the input is given to a CNN, how would we encode these frames to serve as input to the CNN?

As nbro mentioned in a comment to your answer, this question has very recently been asked and answered here.

I also want to know since $s_1$ is equal to a set, which can be seen in line 4 of the algorithm, then why is $s_{t+1}$ equal to $s_t$, $a_t$, $x_{t+1}$?

The algorithm presented in the original DQN paper is relatively simple and written to express the main ideas of their approach (e.g. experience replay, preprocessing histories, gradient descent, etc.); in fact, it isn't even the exact algorithm used in the experiments! For example, the experiments use frame-skipping to reduce computation - this is not mentioned in Algorithm 1 in the paper. With that in mind, setting $s_{t+1}$ equal to $s_t, a_t, x_{t+1}$ in the algorithm signifies a general notion of constructing the next raw state $s_{t+1}$ from the previous preprocessed state $s_t$, previous action $a_t$, and current frame $x_{t+1}$. For example:

• If the action space at the next timestep is constrained by the state, then the state may need additional parameters to encode the action space.
• The algorithm needs some indication if a state is terminal, and such an indication may need to be encoded in the state.
• If there is frame skipping, then multiple frames will be needed to construct the next state, possibly using the previous state as well.

The above examples should display how the encoding of the state cannot always simply be a stack of raw frames, or even a function of $s_t$, $a_t$ and $x_{t+1}$, and therefore a more general approach is often required.

Answer 2

I read the DQN paper titled: Playing Atari with Deep Reinforcement Learning again

I read, in the pre-processing and model architecture section (section 4.1), that for each state that is input to the CNN, that this state is actually stacked frames of the game, so basically what has to be done, to my understanding, is that for each time step you stack 4 frames (current frame and 3 previous frames) and this will serve as input to the CNN as the dimensions would be side * side * 4, 4 because the frames are converted to grey-scale and 4 frames are being used.