What is the "Hello World" problem of Reinforcement Learning?

Question

As we all know, "Hello World" is usually the first program that any programmer learns/implements in any language/framework.

As Aurélien Géron mentioned in his book that MNIST is often called the Hello World of Machine Learning, is there any "Hello World" problem of Reinforcement Learning?

A few candidates that I could think of are multi armed bandits problem and Cart Pole Env.

Answer 1

MNIST (along with CIFAR) may be the "Hello World" of supervised learning for image classification, but it is definitely not the "Hello World" of all machine learning techniques, given that RL is also part of ML and MNIST is definitely not the "Hello World" of RL.

I don't think there is a single "Hello World" problem for RL. However, if you are looking for simple problems (or environments) that are usually used as baselines to assess the quality of RL agents, then I would say that the simple grid worlds where you need to move from one place to the other, the CartPole, MountainCar, Pendulum or other environments listed here are often used.

The environment that you choose to train and test your RL agent depends on your goals. For example, if you designed an algorithm that is supposed to deal with continuous action spaces, then an environment where you can take only a discrete number of actions may not be a good option.

The mentioned environments are very simple (i.e. toy problems). In my opinion, we need more serious environments that can show the applicability of RL to other areas other than (relatively simple) games.

Answer 2

While there's no simple Hello World problem of RL, if your aim is to understand the basic working of Reinforcement Learning and see it at play while using as few moving parts as possible, a simple suggestion would be using Tabular Q-Learning in a toy environment (like your suggested Cart-Pole Env).

Here's the reasoning behind this suggestion

Let's say we interpret MNIST's label as a Hello World of Supervised Learning to mean something showing the basic steps of doing Supervised Learning: Create a model, load the data, then train.

If that interpretation is not far off, we can say a simple introductory problem to Reinforcement Learning (RL) should focus on easily demonstrating a working Markov Decision Process (MDP) which is the backbone of the RL decision making process. As such, this minimal working would involve: Observing the world, selecting an action, as shown in this loop:

This picture is missing two important steps in an RL algorithm learning loop:

1. Estimating the rewards or Fitting the model
2. Improving how you select actions. (Updating your policy)

How we decide to update the policy, or fit the model is what makes difference in the RL algorithm most of the time.

So a suggested first problem would be one that helps you see the MDP in action, while keeping steps 1 and 2 simple enough so that you understand how the agent learns. Tabular Q-Learning seems clear enough for this because it uses a Q-table represented as a 2D array to do the two steps. This should not suggest Q-learning is a "Hello World" RL algorithm because of the said relative ease in understanding it :)

You will be unable to use it's Tabular version anywhere else than in a toy environment though, typically Frozen-Lake and CartPole. An improvement would be using a neural network instead of a table to estimate Q values.

Here are a few useful resources:

1. Q-Learning with Tables (Guide)
2. Q-learning jupyter notebook (Code ~25 lines)
3. Q-Learning with Frozen-Lake and Taxi (Code)
4. Reinforcement Learning with Q-Learning (Guide)

A multi-armed bandit would also be great in introducing you to exploration-exploitation trade-off (which Q-learning does too), though it wouldn't be considered a full RL algorithm since it has no context.