What is curriculum learning in reinforcement learning?

Question

I recently came across the term "curriculum learning" in the context of DRL and was intrigued by its potential to improve the learning process. As such, what is curriculum learning? And how can it be helpful for the convergence of RL algorithms?

Answer 1

Curriculum learning is a general technique for deep learning, which got recently applied to also deep reinforcement learning.

• It's about designing tasks to guide the learning process of the network or agent. This can be particularly useful to bootstrap the network for very hard problems, and even to achieve better convergence.
• In the context of DRL, apart from designing tasks you can also reason on the experiences e.g. by sorting them to facilitate the agent's learning.
• In general the tasks are related to the main problem to solve, but you usually start from very simple ones and gradually increasing the task complexity as the agent makes progresses.

• Indeed, a major issue is how to best design such tasks: there are some works that propose to automate this, giving rise to automated curriculum learning.

For Deep RL I suggest you this survey.

---

Edit: a real-word example of curriculum learning applied to deep RL.

Say you want to solve autonomous driving (AD) with RL. Your environment is a simulator of one or more towns (like CARLA). You can design the curriculum by devising simple variations of your initial env, for example from simpler to harder:

1. Restrict the town to a single straight road without any obstacle (car and pedestrians), in daylight condition, without carrying about complying to traffic rules and speed limits; This is to guide the agent just to learn basic control of the car, and simplified line following.
2. Add cars. This implies to learn collision avoidance: here you want to edit the reward function too, to add a penalty for collisions.
3. Add pedestrians: the collision penalty should be much higher.
4. Driving in more complex scenarios like intersections.
5. Introduce weather conditions and night scenarios. Aim: make the driving policy be more robust.
6. Compliance to traffic rules and speed limits: add terms to the reward fn.
7. Care about time, distance, fuel consumption etc

So you add difficulties until reaching the maximum that corresponds to solving your original environment. Indeed, in doing so you can either edit your env or its reward function. You can even restrict the env to a sub-task: e.g. learning to drive in straight roads.

As written earlier there is no golden rule to best design a curriculum or even to automate its generation (or at least I'm not aware of that.)

The example I provided here is from my own experience (see the paper, chapter F): this was my first time dealing with RL, the results are not great, and the benefit of the curriculum is marginal but I believe is possible to improve that a lot.

Answer 2

Curriculum learning is a training strategy in the context of DRL and other machine learning methods that involves organizing the learning process in a way that gradually increases the complexity of tasks or training samples. It is inspired by the way humans and animals learn, where they start with simple tasks and then progress to more complex ones as their skills develop. The idea is to create a "curriculum" that enables the model to learn more effectively by leveraging its acquired knowledge from easier tasks to solve more difficult ones.

By starting with simpler tasks, the agent can learn the basic skills required to solve the problem more quickly, which can lead to faster convergence compared to being exposed to complex tasks from the beginning.

Update

To answer your questions in the comment: All of the above possibilities; it can be different environments or changing the reward function.

• Environment modification: In this approach, tasks are defined by creating different environments with varying complexity. The complexity can be measured by the number of states, actions, or transitions involved in solving the task. For example, in a robotic manipulation task, you can start with a simple environment where the robot needs to pick up an object at a fixed position, and gradually increase the complexity by adding more objects, changing their positions, or introducing obstacles.
• Subtask decomposition: The main task is split into a sequence of smaller subtasks, which the agent must learn to solve in a specific order. The complexity of each subtask can be measured by the number of intermediate steps required to complete it, the level of abstraction, or the degree of difficulty in achieving the subgoal. By solving subtasks in a specific order, the agent can gradually build the skills necessary to complete the main task.
• Reward shaping: Tasks can be defined by modifying the reward function to guide the agent towards solving specific subtasks or achieving intermediate goals. In this case, the complexity of the task can be determined by the difficulty of obtaining the shaped rewards. As the agent progresses, the reward function can be adjusted to phase out intermediate rewards and focus on the main objective.
• Difficulty parameters: You can define tasks by adjusting certain parameters in the environment that control the difficulty of the problem. For example, in a maze navigation task, the complexity can be adjusted by increasing the size of the maze or adding more obstacles. The tasks can be ordered based on the values of these parameters, with the agent starting with simpler configurations and progressing to more challenging ones.

Each approach is possibly not clearly delineated from the others, changing the maze size when changing difficulty parameters can also be interpreted as changing the environment.