The concept of "planning" is not just related to RL. In general (as the name suggests), planning consists in creating a "plan" which you will use to reach a "goal". The goal depends on the context or problem. For example, in robotics, you can use a "planning algorithm" (e.g. Dijkstra's algorithm) in order to find the path between two points on a map (given e.g. the map as a graph).
In RL, planning usually refers to the use of a model of the environment in order to find a policy that hopefully will help the agent to behave optimally (that is, obtain the highest amount of return or "future cumulative discounted reward"). In RL, the problem (or environment) is usually represented as a Markov Decision Process (MDP). The "model" of the environment (or MDP) refers to the transition probability distribution (and reward function) associated with the MDP. If the transition model (and reward function) is known, you can use an algorithm that exploits it to (directly or indirectly) find a policy. This is the usual meaning of planning in RL. A common planning algorithm in RL is e.g. value iteration (which is a dynamic programming algorithm).
Other researchers have developed theories of planning with general goals, but without considering planning's role in real-time decision- making, or the question of where the predictive models necessary for planning would come from.
Planing is often performed "offline", that is, you "plan" before executing. While you're executing the "plan", you often do not change it. However, often this is not desirable, given that you might need to change the plan because the environment might also have changed. Furthermore, the authors also point out that planning algorithms often have a few limitations: in the case of RL, a "model" of the environment is required to plan.
For example, we have mentioned that much of machine learning research is concerned with supervised learning without explicitly specifying how such an ability would finally be useful.
I think the authors simply want to say that supervised learning is usually used to solve specific problems. The solutions to supervised problems often are not directly applicable to other problems, so this makes them limited.
Another key feature of reinforcement learning is that it explicitly considers the whole problem of a goal-directed agent interacting with an uncertain environment.
In RL, there is the explicit notion of a "goal": there is an agent that interacts with an environment in order to achieve its goal. The goal is often to maximize the "return" (or "future cumulative discounted reward", or, simply, the reward in the long run).
How is RL different from planning and supervised learning?
RL and planning (in RL) are quite related. In RL, the problem is similar to the one in planning (in RL). However, in RL, the transition model and reward function of the MDP (which represents the environment) are usually unknown. Therefore, the only way of finding or estimating an optimal policy that will allow the agent to (near-optimally) behave in this environment is to interact with the environment and gather some info regarding its "dynamics".
RL and supervised learning (SL) are quite different. In SL, there isn't usually the explicit concept of "agent" or "environment" (and their interaction), even though it might be possible to describe supervised learning in that way (see this question). In supervised learning, during the training or learning phase, a set of inputs and the associated expected outputs is often provided. Then the "objective" is to find a map between inputs and outputs, which generalizes to inputs (and corresponding outputs) that have not been observed during the learning phase. In RL, there isn't such a set of inputs and associated expected outputs. In RL, there is just a scalar signal emitted by the environment, at each time step, which roughly indicates how well the agent is currently performing. However, the goal of the agent is not just to obtain rewards, but to behave optimally (in the long run).
In short, in RL, there is the explicit notion of agent, environment and goal, and the reward is the only signal which tells the agent how well it is performing, but the reward does not tell the agent which actions it should take at each time step. In supervised learning, the objective is to find a function that maps inputs to the corresponding outputs, and this function is learned by providing explicit examples of such mappings during the training phase.
There are some RL algorithms (like the temporal-difference ones), which could roughly be thought of as self-supervised learning algorithms, where the agent learns from itself (or from the experience it has gained by interacting with the environment). However, even in these cases, the actions that the agent needs to take are not explicitly taught.
