I am watching DeepMind's video lecture series on reinforcement learning, and when I was watching the video of model-free RL, the instructor said the Monte Carlo methods have less bias than temporal-difference methods. I understood the reasoning behind that, but I wanted to know what one means when they refer to bias-variance tradeoff in RL.

Is bias-variance trade-off used in the same way as in machine learning or deep learning?

(I am just a beginner and have just started learning RL, so I apologize if it is a silly question.)


1 Answer 1


The bias-variance trade-off that you're referring to has to do with the return estimator. Any RL algorithm you choose needs some estimate of the cumulative return, which is a random variable with many sources of randomness, such as stochastic transitions or rewards.

Monte Carlo RL algorithms estimate returns by running full trajectories and literally averaging the return achieved for each state. This imposes very few assumptions on the system (in fact, you don't even need the Markovian property for these methods), so bias is low. However, variance is high since each estimate depends on the literal trajectories that you observe. As such, you'll need many, many trajectories to get a good estimate of the value function.

On the other hand, with TD methods, you estimate returns as $R_t + \gamma V(S_{t+1})$, where $V$ is your estimate of the value function. Using $V$ this imposes some bias (for instance, the initialization of the value function at the beginning of training affects your next value function estimates), with the benefit of reducing variance. In TD learning, you don't need full environment rollouts to make a return estimate, you just need one transition. This also lets you make much better use of what you've learned about the value function, because you're learning how to infer value "piecewise" rather than just via literal trajectories that you happened to witness.

  • $\begingroup$ It may also be a good idea to mention TD($\lambda$), so that people don't think that the distinction between the 2 approaches is always sharp. $\endgroup$
    – nbro
    Commented Dec 19, 2021 at 20:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .