3
$\begingroup$

In reinforcement learning approaches, like temporal-difference (TD) learning or Monte Carlo methods, two of the metrics used to measure their performance are the bias and the variance.

What do these terms mean? Which characteristic of the agent do they affect (value functions, policy, optimal behavior, etc.)?

$\endgroup$
4
  • 1
    $\begingroup$ Related: stats.stackexchange.com/questions/364997/… $\endgroup$ – Neil Slater Jul 25 '19 at 8:30
  • $\begingroup$ @NeilSlater you have said in your answer that in TD learning the values are biased to the initial values. Does this mean if we use random initialization the bias will be low? Since now every state is depending on some random value of the next state thus has lesser bias and the earlier state depending on this state and hence has lesser bias too (this as compared to initialization of all states with 0/fixed value)? $\endgroup$ – user9947 Jul 25 '19 at 8:41
  • $\begingroup$ It might reduce the bias, but you won't know that a priori. The only way of reducing the bias in TD learning consistently is to start with known good estimates. E.g. set initial Q values using some kind of heuristic that you know in some cases is close to the true optimal value function, and in worst cases is no worse than some default value like zero. Generally what you hope is that the bias in TD learning will reduce over time due to updates - for tabular learning this is guaranteed $\endgroup$ – Neil Slater Jul 25 '19 at 9:33
  • $\begingroup$ Although it's a newer question, this seems to be a duplicate (although not exact) of your question. I will leave this question open for a while because I think it may be interesting to get a canonical answer that defines bias and variance mathematically in the context of reinforcement learning. $\endgroup$ – nbro Feb 5 at 10:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy