What is the difference between imitation learning and classification done by experts?

Question

In short, imitation learning means learning from the experts. Suppose I have a dataset with labels based on the actions of experts. I use a simple binary classifier algorithm to assess whether it is good expert action or bad expert action.

How is this binary classification different from imitation learning?

Imitation learning is associated with reinforcement learning, but, in this case, it looks more like a basic classification problem to me.

What is the difference between imitation learning and classification done by experts?

I am getting confused because imitation learning relates to reinforcement learning while classification relates to supervised learning.

Answer 1

Imitation learning is supervised learning applied to the RL setting.

In any general RL algorithm (such as Q-learning), the learning is done on the basis of the reward function. However, consider a scenario where you have available the optimal policy in the form of a table, mapping each state to each action. In this scenario you will not care about the rewards received - rather, you'd simply do a table lookup to decide the optimal action.

This scenario is impractical in most settings because the table for the optimal policy will be too big. However, if you have enough entries from the table, you can use a general function approximator such as a neural network to find the optimal action. Again, you do not need to look at the rewards, but only at the state $\rightarrow$ action mappings. I do not know imitation learning in detail beyond this, but I suspect in the case of discrete actions (such as in Chess, Go), it would be trained with a cross-entropy objective as is typical of classification tasks.

I suspect that the reason this has a different name in the RL setting is because this is different than how the conventional RL algorithms work. Also, much of RL thinking is inspired from everyday interaction / biology. Think of how we learn how to drive, or play sports such as soccer. Typically there is a coach who tells you what actions to take under different conditions, and you imitate those actions.

Answer 2

I also had the same question, but after looking at this two links: this article and this lecture I think we can say that behavioral cloning (which is the simplest way for doing imitation learning) is just normal supervised learning. But imitation learning could be associated with RL because it has other types (improvements) that involve learning while interacting with the environment like Dataset Aggregation, where you have an interactive expert that you can query for optimal actions to label observations obtained by rolling out the learned policy in the environment, and you repeat collecting the data and training the policy. This is done to mitigate the known problem of distributional drift in BC. More about it can be found here.

Even with this interactive expert, I still think it is supervised learning, but I can understand why some people might associate it with RL.