# What is the bias-variance trade-off in reinforcement learning?

I am watching DeepMind's video lecture series on reinforcement learning, and when I was watching the video of model-free RL, the instructor said the Monte Carlo methods have less bias than temporal-difference methods. I understood the reasoning behind that, but I wanted to know what one means when they refer to bias-variance tradeoff in RL.

Is bias-variance trade-off used in the same way as in machine learning or deep learning?

(I am just a beginner and have just started learning RL, so I apologize if it is a silly question.)

On the other hand, with TD methods, you estimate returns as $$R_t + \gamma V(S_{t+1})$$, where $$V$$ is your estimate of the value function. Using $$V$$ this imposes some bias (for instance, the initialization of the value function at the beginning of training affects your next value function estimates), with the benefit of reducing variance. In TD learning, you don't need full environment rollouts to make a return estimate, you just need one transition. This also lets you make much better use of what you've learned about the value function, because you're learning how to infer value "piecewise" rather than just via literal trajectories that you happened to witness.
• It may also be a good idea to mention TD($\lambda$), so that people don't think that the distinction between the 2 approaches is always sharp.