Why use semi-gradient instead of full gradient in RL problems, when using function approximation?

Question

Semi-gradient methods work well in reinforcement learning, but what is there a reason of not using the true gradient if it can be computed?

I tried it on the cart pole problem with a deep Q-network and it performed much worse than traditional semi-gradient. Is there a concrete reason for this?

Answer 1

Just so that this could be useful for people who refer to this post later on: Please refer to Sutton's reinforcement learning book (2nd edition) example 11.2. It provides an example for why full gradient wouldn't work.

Answer 2

Semi gradient methods work well in Reinforcement Learning, but what is the reason of not using the true gradient if it can be computed?

Just complexity and extra computation, in many cases for a marginal benefit.

I tried it on the cart pole problem with a deep Q-Network and it performed much worse than traditional semi gradient, is there a concrete reason for this?

It is hard to tell, without exploring the implementation in detail. However, DQN is an inherently unstable learning technique that needs care in choosing hyper-parameters that control this instability and offset against learning rate:

• size of minibatch to train from experience replay on each step
• number of training steps between taking frozen copies for estimation*
• whether or not you use double-learning to avoid maximisation bias (more important if you have fine-grained discretisation of continuous action space)

There is a chance that the optimal choices here are different between true gradient and semi gradient approaches.

* The frozen estimator could be a big clue here in your implementation. If you are using this frozen copy technique, it has a big impact on how you should calculate the true gradient, because changing the parameters would no longer change the current TD target - which is what the true gradient approach fixes. However, getting rid of this stability-improving addition in order to get true gradients might on balance make the algorithm less stable - you could try to fix that by taking larger mini-batches.