How to implement RAM versions of Atari games

Question

I have coded the breakout RAM version, but, unfortunately, its highest reward was 5. I trained it for about 2 hours and never reached a higher score. The code is huge, so I can't paste here, but, in short, I used double deep Q-learning, and trained it like it was CartPole or lunar-lander environment. In CartPole, the observation was a vector of 4 components. In that case, my double deep Q-learning agent solved the environment, but in the breakout-ram version whose observation was a vector of 128 elements, it was not even close.

Did I miss something?

Answer 1

To my knowledge from reading about model-based and model-free reinforcement learning,

DQN and Double DQN are model-free reinforcement learning methods. (Why am I mentioning this, see below):

https://link.springer.com/referenceworkentry/10.1007%2F978-1-4614-7320-6_674-1

You should see under the heading "Definition" (you may need to scroll down in the web-page) in the web-page (can be accessed by the above link), it states that: "model-free techniques require extensive experience."

Now extensive experience, depending of how fast you can go through states, can take a few days even weeks...model-free methods require a lot of samples to learn.

Also there are a lot of states: there are ($256^{128}$) states. That is a really big number (I'm just emphasizing that training may take a long while)

Due implementation specifics not being supplied in question, I am assuming your implementation is correct, although...you are using RAM. DQN used image data, I very quickly skimmed the Double DQN paper: https://arxiv.org/pdf/1509.06461.pdf . I am assuming they used image data as well because Double DQN was compared to DQN.