# Which Openai Gym environment should I use to test a Temporal Difference RL Agent?

Up to now, I have been using (my version of) open AI's code, with the suggested CartPole.

I have been using Monte Carlo methods, which, for cartpole, seemed to work fine.

Trying to move to temporal difference, Cartpole seems to fail to learn (with simple TD method) (or I stopped it too soon, but still unacceptable).

I assume that is the case because in Cartpole, for every timestamp, we get a reward of 1, which has very little immediate information about weather or not the action was good or not.

Which gym environment is the simplest that would probably work with TD learning?

by simplest I mean that there is no need for a large NN to solve it. No conv nets, no RNNS. just a few small layers of a fully connected NN, just like in cartpole. something I can train on my home cpu, just to see it starting to converge.

Cartpole will work just fine with a single-step TD method - e.g. Q learning - and a simple neural network. Configured well, a simple DQN model should solve it in a few minutes. You can also forgo the neural network and use discretised states, or tile coding etc.

What you cannot do is plug a neural network estimator into basic Q learning and make no other adjustments. That is because the bootstrap process in TD learning will create a runaway feedback driven from the random and incorrect return estimates in the initialised NN. You have to use something like DQN to get it to work at all. This is different to Monte Carlo approach, where you can generally just plug in a NN in place of a Q table.

The basics of DQN are:

• Use experience replay. Store $$(S, A, R, S')$$ values from each step, and when training the NN, draw a small random batch (e.g. maybe 16 samples) from this table, re-estimate their Q values to calculate TD target e.g. $$R + \gamma\text{max}_{a'} Q(S',a')$$ and use that to create a minibatch to train the NN

• Use a "delayed" target network for estimating TD target. This can just be a snapshot of your learning network, taken once every N steps, where N is typically anything from 100 to 100000.

Further details of what configured well means varies from problem to problem. With DQN you will want to play with hyper parameters of experience history size, time to keep frozen copy of network (for bootstrap estimates), batch size of experience table samples, amount of exploration.

In CartPole though, there will be a wide range of acceptable hyperparameter values. So I suspect you have just plugged in a NN in place of the Q table and wonder why it doesn't work.

I assume that is the case because in Cartpole, for every timestamp, we get a reward of 1, which has very little immediate information about weather or not the action was good or not.

Yes this is part of what makes some learning goals become harder challenges in sequential-decision control systems. However, this was also the case for your Monte Carlo method, and in general this relates to the credit assignment problem in RL.

• Thanks :) I will go on to implementing what you suggested – Gulzar Feb 2 at 17:26
• Also, why does having the "delayed" target network help stability? after all, we take an action, and receive a reward. Why does it matter if that action and reward came from a (temporarily) constant distribution or not? – Gulzar Feb 2 at 18:31
• Also by the way, When drawing an experience whose s is terminal (meaning no s'), should I just plug in 0 (zero) in place of Q(s',a')? – Gulzar Feb 2 at 18:44
• @Guizar: Yes, plug in a zero, the value of $Q(s_T, \cdot)$ is 0 by definition when $s_T$ is a terminal state, because Q is predicting future rewards and there are none. The delayed network is a separate question, too much detail for a comment - I think I have answered a similar one here on the site, sorry cannot find it at the moment – Neil Slater Feb 2 at 19:47