In algorithms like MC/TD (tabular value approximation) two of the metrics used to measure their performance are the bias and the variance. What do these terms mean? And which characteristic of the agent do they affect (value functions, policy, optimal behaviour, etc)?

  • 1
    $\begingroup$ Related: stats.stackexchange.com/questions/364997/… $\endgroup$ – Neil Slater Jul 25 '19 at 8:30
  • $\begingroup$ @NeilSlater you have said in your answer that in TD learning the values are biased to the initial values. Does this mean if we use random initialization the bias will be low? Since now every state is depending on some random value of the next state thus has lesser bias and the earlier state depending on this state and hence has lesser bias too (this as compared to initialization of all states with 0/fixed value)? $\endgroup$ – DuttaA Jul 25 '19 at 8:41
  • $\begingroup$ It might reduce the bias, but you won't know that a priori. The only way of reducing the bias in TD learning consistently is to start with known good estimates. E.g. set initial Q values using some kind of heuristic that you know in some cases is close to the true optimal value function, and in worst cases is no worse than some default value like zero. Generally what you hope is that the bias in TD learning will reduce over time due to updates - for tabular learning this is guaranteed $\endgroup$ – Neil Slater Jul 25 '19 at 9:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.