I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in short, I used double deep Q-learning, and trained it like it was CartPole or lunar-lander environment. In CartPole, the observation was a vector of 4 components. In that case, my double deep Q-learning agent solved the environment, but in the breakout-ram version whose observation was a vector of 128 elements, it was not even close.

Did I miss something?

  • $\begingroup$ It does sound like something's off. Have you already used aggregates of multiple frames so that the movement of the ball and paddle are captured? My own solutions to this a few years back easily achieved perfect/infinite games after a few hours. $\endgroup$ Commented Nov 21, 2023 at 16:37

1 Answer 1


To my knowledge from reading about model-based and model-free reinforcement learning,

DQN and Double DQN are model-free reinforcement learning methods. (Why am I mentioning this, see below):


You should see under the heading "Definition" (you may need to scroll down in the web-page) in the web-page (can be accessed by the above link), it states that: "model-free techniques require extensive experience."

Now extensive experience, depending of how fast you can go through states, can take a few days even weeks...model-free methods require a lot of samples to learn.

Also there are a lot of states: there are ($256^{128}$) states. That is a really big number (I'm just emphasizing that training may take a long while)

Due implementation specifics not being supplied in question, I am assuming your implementation is correct, although...you are using RAM. DQN used image data, I very quickly skimmed the Double DQN paper: https://arxiv.org/pdf/1509.06461.pdf . I am assuming they used image data as well because Double DQN was compared to DQN.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .