Consider a problem where we have a finite number of states and actions. Given a state and an action, we can easily know the reward and the next state (deterministic). The state space is really large and has no final state.

There was a paper that for a problem of this type used TD(0) by filling the value table and chose its actions by:

$$\pi(s) = \text{argmax}_a (r(s,a) + \gamma V(s_{(t+1)}))$$

I've read somewhere that is OK to use prediction algorithms when the model is well-described with the objective of choosing best actions and not only evaluating the policy.

What is the purpose, advantages and disadvantages of using TD prediction here instead of a TD control algorithms (and just saving the $Q(s, a)$ table)? If it was about space, you still have to store a table with all the rewards for each state-action pair, right?

I'm not sure if I was able to explain myself very well as I was trying to keep it short, if some clarification is needed please tell me.

  • 1
    $\begingroup$ Hi Miguel. I have edited your question to hopefully make it clearer. Have I changed the meaning of the question? If yes, feel free to edit it again or roll it back to the previous version. $\endgroup$ – nbro May 13 '19 at 17:37
  • $\begingroup$ Yep, it's good thank you :) $\endgroup$ – Miguel Saraiva May 13 '19 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.