RL vs Supervised Learning vs Planning

This is an excerpt taken from Sutton and Barto (pg. 3):

Another key feature of reinforcement learning is that it explicitly considers the whole problem of a goal-directed agent interacting with an uncertain environment. This is in contrast with many approaches that address subproblems without addressing how they might fit into a larger picture. For example, we have mentioned that much of machine learning research is concerned with supervised learning without explicitly specifying how such an ability would finally be useful. Other researchers have developed theories of planning with general goals, but without considering planning's role in real-time decision- making, or the question of where the predictive models necessary for planning would come from. Although these approaches have yielded many useful results, their focus on isolated subproblems is a significant limitation.

I have an idea of Supervised Learning, but what exactly does the author mean by Planning? And how is the RL approach different from planning and Supervised Learning?

(Illustration with an example would be nice).

The concept of "planning" is not just related to RL. In general (as the name suggests), planning consists in creating a "plan" which you will use to reach a "goal". The goal depends on the context or problem. For example, in robotics, you can use a "planning algorithm" (e.g. Dijkstra's algorithm) in order to find the path between two points on a map (given e.g. the map as a graph).

In RL, planning usually refers to the use of a model of the environment in order to find a policy which hopefully will help the agent to behave optimality (that is, obtain the highest amount of return or "future cumulative discounted reward"). In RL, the problem (or environment) is usually represented as a Markov Decision Process (MDP). The "model" of the environment (or MDP) refers to the transition probability distribution (and reward function) associated with the MDP. If transition model (and reward function) is known, you can use an algorithm which exploits it to (directly or indirectly) find a policy. This is the usual meaning of planning in RL. A common planning algorithm in RL is e.g. value iteration (which is a dynamic programming algorithm).

Other researchers have developed theories of planning with general goals, but without considering planning's role in real-time decision- making, or the question of where the predictive models necessary for planning would come from.

Planing is often performed "offline", that is, you "plan" before executing. While you're executing the "plan", you often do not change it. However, often this is not desirable, given that you might need to change the plan because the environment might also have changed. Furthermore, the authors also point out that planning algorithms often have a few limitations: in the case of RL, a "model" of the environment is required to plan.

For example, we have mentioned that much of machine learning research is concerned with supervised learning without explicitly specifying how such an ability would finally be useful.

I think the authors simply want to say that supervised learning is usually used to solve specific problems. The solutions to supervised problems often are not directly applicable to other problems, so this makes them limited.

Another key feature of reinforcement learning is that it explicitly considers the whole problem of a goal-directed agent interacting with an uncertain environment.

In RL, there is the explicit notion of a "goal": there is an agent which interacts with an environment in order to achieve its goal. The goal is often to maximise the "return" (or "future cumulative discounted reward", or, simply, the reward in the long run).

How is RL different from planning and supervised learning?

RL and planning (in RL) are quite related. In RL, the problem is similar to the one in planning (in RL). However, in RL, the transition model or reward function of the MDP (which represents the environment) is unknown. Therefore, the only way of finding or estimating a (hopefully, optimal) policy which will allow the agent to (optimally) behave in this environment is to interact with the environment and gather some info regarding its "dynamics".

RL and supervised learning are quite different. In supervised learning, there isn't usually the explicit concept of "agent" or "environment" (and their interaction), even though it might be possible to describe supervised learning in that way. In supervised learning, during the training or learning phase, a set of inputs and the associated expected outputs is often provided. Then the "objective" is to find a map between inputs and outputs, which generalises to inputs (and corresponding outputs) which have not been observed during the learning phase. In RL, there isn't such a set of inputs and associated expected outputs. In RL, there is just scalar signal emitted by the environment, at each time step, which roughly indicates how well the agent is currently performing. However, the goal of the agent is not just to obtain rewards, but to behave optimally (in the long run).

In short, in RL, there is the explicit notion of agent, environment and goal, and the reward is the only signal which tells the agent how well it is performing, but the reward does not tell the agent which actions it should take at each time step. In supervised learning, the objective is to find a function which maps inputs to the corresponding outputs, and this function is learned by providing explicit examples of such mappings during the training phase.

There are some RL algorithms (like the temporal-difference ones), which could roughly be thought of as self-supervised learning algorithms, where the agent learns from itself (or from the experience it has gained by interacting with environment). However, even in these cases, the actions that the agent needs to take are not explicitly taught.

The automated planning is:

Automated planning and scheduling, sometimes denoted as simply AI Planning,1 is a branch of artificial intelligence that concerns the realization of strategies or action sequences, typically for execution by intelligent agents, autonomous robots and unmanned vehicles. Unlike classical control and classification problems, the solutions are complex and must be discovered and optimized in multidimensional space. Planning is also related to decision theory.

In known environments with available models, planning can be done offline. Solutions can be found and evaluated prior to execution. In dynamically unknown environments, the strategy often needs to be revised online. Models and policies must be adapted. Solutions usually resort to iterative trial and error processes commonly seen in artificial intelligence. These include dynamic programming, reinforcement learning and combinatorial optimization. Languages used to describe planning and scheduling are often called action languages.

In other words, the planning is some strategies or actions to reach from the start state to the goal state. As you found in the above, one of the solutions for planning could be RL (depends on the problem). Hence, MDP is a specific case of planning and it is more general.

For the difference of RL and supervised learning you can see this post:

The main difference is to do with how "correct" or optimal results are learned:

• In Supervised Learning, the learning model is presented with an input and desired output. It learns by example.

• In Reinforcement Learning, the learning agent is presented with an environment and must guess correct output. Whilst it receives feedback on how good its guess was, it is never told the correct output (and in addition the feedback may be delayed). It learns by exploration, or trial and error.

• I was actually looking for the difference between planning and RL. – DuttaA Feb 16 '19 at 16:58
• @DuttaA as I wrote: the planning is some strategies or actions to reach from the start state to the goal state. As you found in the above, one of the solutions for planning could be RL (depends on the problem). – OmG Feb 16 '19 at 16:59