1
$\begingroup$

I am writing a couple of different reinforcement learning models based on Rainbow DQN or some PG models. All of them internally use an LSTM network because my project is using time series data.

I wanted to test my models using OpenAI Gym before I add too many domain specific code to the models.

The problem is that, all of the Atari games seem to fall into the CNN area which I don't use.

Is it possible to use OpenAI Gym to test any time series data driven RL models/networks?

If not, is there any good environment that I can use to examine the validity of my models?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

If I understand your problem correctly, you can test on just about any environment, and just omit parts of the observations to ensure your RNN is learning. For example, you can test on cartpole, ignoring the velocity and angular velocity states. This way the MDP isn't actually Markovian and you'll need the RNN to learn.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Hi harwiltz, appreciate your quick reply. I am in fact not familiar with the Gym environment and didn't know there are observations like velocity. My understanding on Gym comes from reading a few well cited paper such as the one for Rainbow DQN where they seem to use the game frame images as input and hence using CNN as the internal network. I don't know how I can feed such environment data to a LSTM/RNN instead. Or do I not have to use frame image data as input? $\endgroup$
    – ZXY
    Jun 21, 2020 at 16:44
  • $\begingroup$ Not all gym environments have pixel state spaces. If you want to train from pixels, you should probably make a CNN feature extractor and pass the extracted featured to your RNN. If you just want to test your network though, I'd recommend training on simpler environments (look up cartpole, lunarlander, they're both in openai gym for example), whose state spaces are basically just position, velocity, etc, and discard some dimensions to force your RNN to learn. $\endgroup$
    – harwiltz
    Jun 21, 2020 at 18:35
  • $\begingroup$ I think I get the picture. Will go and do some study. Thanks. Last question, if I only use data like position, velocity etc as state space for training, will I still be able to get good testing results? After all, getting good Atari game results is how I would tell if my own model is good or not. $\endgroup$
    – ZXY
    Jun 21, 2020 at 23:08
  • $\begingroup$ With Atari, you can only get pixel or RAM states, both of which you'll probably need to implement a feature extractor to preprocess your RNN inputs. Otherwise, when you have access to states like position and velocity and such, that generally makes the problem much easier (the data will have much lower dimension). $\endgroup$
    – harwiltz
    Jun 22, 2020 at 4:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .