2
$\begingroup$

In all the tutorials of deep Q-learning (using neural networks) I have read so far, the state-action value function $Q(s,a)$ is learned by temporal difference learning. However, in policy gradient methods, it is also possible to learn by using Monte-Carlo estimate of returns (i.e. observing all the rewards in a trajectory), if the task is episodic.

I would like to know if it is also possible to do deep Q-learning by using Monte-Carlo estimate of returns in episodic task?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Technically, it would not be Q-learning if you used Monte Carlo returns, because Q-learning is specifically a temporal difference (TD) method.

However, your intuition about the estimates being interchangeable is correct. There is nothing preventing you in principle from swapping out the TD return estimate for a Monte Carlo sample to generate data for training a neural network that is approximating $Q(s, a)$. In some ways it is a better estimate, as it will be unbiased. If you stop using bootstrapping, then the learning process is theoretically more stable and less likely to diverge.

There are some caveats:

  • Experience replay becomes less practical. You could store the final return from each $(s,a)$ pair, but it will quickly become off-policy to a degree that it could not be used. Some extensions to DQN do use a small n with n-step returns and no correction. However, you have to be careful with this, and full trajectories off-policy would need better correction - many would be useless, especially at the start of learning.

  • Without experience replay, you will need more samples of the current policy output before training. Training on a batch generated from just a single trajectory could easily result in poor learning performance, because records would be closely correlated with each other. So you would need to borrow ideas from CEM or A2C where you generate many trajectories for each training batch.

These issues could make a Monte-Carlo based deep RL slower to train than DQN. However, perhaps the extra stability of the approach may appeal for a specific problem.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the reply! But I don't quite understand why is "Q-learning is specifically a TD method"? I mean, what characteristic of Q-learning make it a TD method? A second question is what is the "corrections" you mentioned when using a n-step return? Is there a paper talking about the correction method? $\endgroup$
    – Cloudy
    Commented Mar 24, 2022 at 11:52
  • $\begingroup$ Q-learning uses TD targets as value estimates, that makes it a TD method. The equivalent MC method is called "off-policy Monte Carlo control", it is not called "Q-learning with MC return estmates", although it could be in principle that's not how the original designers of Q-learning chose to categorise what they created. For corrections required for n-step returns see Sutton & Barto chapters on off-policy Monte Carlo. For the n-step augmentation in DQN (which deliberately ignores those corrections and "gets away with it"), see the "Rainbow" paper: arxiv.org/abs/1710.02298 $\endgroup$
    – Neil Slater
    Commented Mar 24, 2022 at 13:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .