Some RL literature use terms such as: 'Bellman backup' and 'Bellman error'. What do these terms refer to?

  • $\begingroup$ There's already an answer that addresses both concerns/questions, but, please, next time, focus on one question per post, although, in this case, the terms are highly related (but I still think these "simple" questions could have been asked in separate posts). It may also be a good idea to provide more context (e.g. a link to an article that mentions these terms), although, again, in this case, anyone familiar with RL would be able to understand the question. $\endgroup$
    – nbro
    Jun 28 at 13:15

A Bellman backup is an application of a Bellman operator. For example, the step

$$ V(x)\leftarrow \alpha(R + \mathbf{E}[V(x')]) + (1-\alpha)V(x) $$

Is a Bellman backup for some learning rate $\alpha$.

A Bellman error is

$$ d(V(x), R + \mathbf{E}[V(x')]) $$

for some metric $d$, usually $d(x, y) = (x-y)^2$.

  • $\begingroup$ What does 'backup' refer to here? $\endgroup$
    – user529295
    Jun 28 at 12:55
  • 1
    $\begingroup$ It refers to propagating information from later states to earlier ones (backward in time sorta) $\endgroup$
    – harwiltz
    Jun 28 at 13:02
  • 2
    $\begingroup$ It may be a good idea to 1. also provide the figures of the backups e.g. that you can find in Sutton and Barto's book, 2. to link the OP to this question about what the Bellman operator is and 3. explain the symbols in your answer. $\endgroup$
    – nbro
    Jun 28 at 13:12
  • $\begingroup$ @nbro Thanks for the references. $\endgroup$
    – user529295
    Jun 28 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.