1
$\begingroup$

In short, imitation learning means learning from the experts. Suppose I have a dataset with labels based on actions of experts. I use a simple binary classifier algorithm to assess whether it is good expert action or bad expert action.

How is this binary classifier different from imitation learning? Imitation learning is associated with reinforcement learning but in this case, it looks more like a basic classification problem to me.

What is the difference between imitation learning and classification done by experts? I am getting confused because imitation learning relates to reinforcement learning while classification relates to supervised learning.

$\endgroup$
1
$\begingroup$

Imitation learning is supervised learning applied to the RL setting.

In any general RL algorithm (such as Q-learning), the learning is done on the basis of the reward function. However, consider a scenario where you have available the optimal policy in the form of a table, mapping each state to each action. In this scenario you will not care about the rewards received - rather, you'd simply do a table lookup to decide the optimal action.

This scenario is impractical in most settings because the table for the optimal policy will be too big. However, if you have enough entries from the table, you can use a general function approximator such as a neural network to find the optimal action. Again, you do not need to look at the rewards, but only at the state $\rightarrow$ action mappings. I do not know imitation learning in detail beyond this, but I suspect in the case of discrete actions (such as in Chess, Go), it would be trained with a cross-entropy objective as is typical of classification tasks.

I suspect that the reason this has a different name in the RL setting is because this is different than how the conventional RL algorithms work. Also, much of RL thinking is inspired from everyday interaction / biology. Think of how we learn how to drive, or play sports such as soccer. Typically there is a coach who tells you what actions to take under different conditions, and you imitate those actions.

$\endgroup$
  • $\begingroup$ Mapping the state-action pairs of a q-table to a neural network is called reinforcement learning. It's not imitating a human, but it's a mathematical driven algorithm to search for a minimum in the error function. The ability to maximize the reward is a built-in feature of any optimization algorithm and works great without observing human actions. $\endgroup$ – Manuel Rodriguez Dec 19 '18 at 9:02
  • $\begingroup$ @ManuelRodriguez That's already pretty much what Sabyasachi wrote in their answer. The answer already describes that in "regular" RL you'd learn a mapping $(s, a) \rightarrow Q(s, a)$, whereas in imitation learning you're already given some sort of (partial) mapping $s \rightarrow a$ (i.e. a policy) and subsequently learn to approximate that directly, without any value estimates. $\endgroup$ – Dennis Soemers Dec 19 '18 at 9:27
  • $\begingroup$ @DennisSoemers Nope, Imitation learning isn't a partial mapping nor an approximation technique. Imitation learning is located outside of mathematics in the domain of “linguistics”. It can't be described as a algebra function. $\endgroup$ – Manuel Rodriguez Dec 19 '18 at 9:33
  • $\begingroup$ @ManuelRodriguez In the context of RL (and that is the context this question is placed in), the explanation in this answer certainly seems like a reasonable explanation for one way to do imitation learning. I'm not 100% sure if it's the only/best way, probably not (I mostly have experience in "traditional" RL personally)... but it's a reasonable explanation. See for example some of the papers listed here: spinningup.openai.com/en/latest/spinningup/… $\endgroup$ – Dennis Soemers Dec 19 '18 at 10:05
  • $\begingroup$ @ManuelRodriguez and in fact it's actually really similar to what you proceeded to write as an answer yourself afterwards. You're saying a trajectory of expert actions is provided as an example. Well, such a trajectory can be viewed directly as a partial policy, a partial mapping from states to actions. $\endgroup$ – Dennis Soemers Dec 19 '18 at 10:07
1
$\begingroup$

The main problem in robotics control is that the state space is to large for an uninformed search algorithm. There are millions of possible robot movements and testing out all would take too long. Imitation learning uses domain knowledge to reduce the state-space. A previous demonstration from an expert guides the policy search. "Imitation" replicates existing domain knowledge and avoids a brute force search in the state space.

The relationship to a labeled dataset is, that such a dataset contains also expert knowledge. An expert has recognized what's on the image, and this information is used as a heuristics. The difference is, that imitation learning is mostly used if action sequences are executed on a robot. The actions over a horizon of 1 minute gets imitated. In contrast, dataset labeling doesn't need time-based information.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.