5
$\begingroup$

What is the difference between the term 'prediction'/value estimation in RL as compared to the 'control' problem?

Are there scenarios in RL where the problem cannot be distinctly categorised into the aforementioned problems and is a mixture of the problems?

Examples where the problem cannot be easily categorised into one of the aforementioned problems would be nice.

$\endgroup$
5
$\begingroup$

Prediction is the problem of predicting any feature of the environment. In reinforcement learning, the typical feature is the reward or return, but this doesn't have to be always the case. See Multi-timescale nexting in a reinforcement learning robot (2011) by Joseph Modayil et al.

Control is the problem of estimating a policy. Clearly, the term control is related to control theory. In fact, the term control is often used as a synonym for action. See Is there any difference between a control and an action in reinforcement learning?. Similarly, the term controller is also used as a synonym for policy. For example, in the paper Metacontrol for Adaptive Imagination-Based Optimization (2017) by Jessica B. Hamrick et al., the terms controller and meta-controller are used to refer to policies and meta-policies, respectively. A controlled system can also refer to the environment.

Section 14.1 of the book Reinforcement learning: an introduction (2nd edition) provides more details about the distinction between prediction and control and how this distinction is related to psychological concepts.

| improve this answer | |
$\endgroup$
5
$\begingroup$

Nbro's answer already addresses the basic definitions, so I won't repeat that. Instead I'll try to elaborate a bit on the other parts of the question.

Are there scenarios in RL where the problem cannot be distinctly categorised into the aforementioned problems and is a mixture of the problems?

I'm not sure about cases where the "problem" can't be distinctly categories... but often, when we're actually interested in control as a problem, we still also actually deal with the prediction problem as a part of our training algorithm. Think of $Q$-learning, Sarsa, and all kinds of other algorithms related to the idea of "Generalized Policy Iteration". Many of them work (roughly) like this:

  1. Initialise (somehow, possibly randomly) a value function
  2. Express a policy in terms of that value function (greedy, $\epsilon$-greedy, etc.)
  3. Generate experience using that policy
  4. Train the value function to be more accurate for that policy (prediction problem here)
  5. Go back to step 2 (control problem here)

You could view these techniques in this way, as handling both of the problems at the same time, but there's also something to be said for the argument that they're really mostly just tackling the prediction problem. That's where all the "interesting" learning happens. The solution to the control problem is directly derived from the solution to the prediction problem in a single, small step. There are different algorithms, such as Policy Gradient methods, that directly aim to address the control problem instead.


An interesting (in my opinion :)) tangent is that in some problems, one of these problems may be significantly easier than the other, and this can be important to inform your selection of algorithm. For example, suppose you have a very long "road" where you can only move to the left or the right, you start on the left, and the goal is all the way to the right. In this problem, a solution to the control problem is trivial to express; just always go right. For the prediction problem, you need something much more powerful to be able to express all the predictions of values in all possible states.

In other problems, it may be much easier to quickly get an estimate of the value, but much more complicated to actually express how to obtain that value. For example, in StarCraft, if you have a much larger army, it is easy to predict that you will win. But you will still need to execute some very specific, long sequences of actions to achieve that goal.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.