Why not use only expert demonstrations in Imitation Learning approaches?

Question

Some IL approaches train the agents by using some specific ratio of expert demonstrations to trajectories generated using the policy being optimized.

In the specific paper I'm reading they say "we experimented with various IL proportions (10-50% by increments of 10%) and observed that the RL/IL ratio does not seem to affect the performance of the trained policy by much."

My question is: why not rely only on expert demonstrations instead of introducing the noise of trajectories generated by a sub-optimal policy?.

My assumptions are:

1. This noise helps explore the state space beyond the specific episodes experienced by the expert system.
2. You might have expert systems that only work on a limited state space and do not scale to bigger, more complex environments. Therefore, although you can't use expert demonstrations in the bigger envs you still want to leverage their experience by learning their behavior in limited settings and the way to avoid overfitting to a specific, constrained policy is by always having some proportion of episodes generated by your policy being learned.

Answer 1

Using suboptimal demonstrations in Imitation Learning

The paper you link above uses Behavior Cloning (supervised learning on the expert state-action pairs), which is known to suffer from the "DAgger problem" like many other imitation learning algorithms.

In essence, a policy trained under imitation learning from expert demonstrations will inevitably make some suboptimal actions during the episode, and these suboptimal actions lead to states that are increasingly different from states in the expert demonstrations, which leads to more suboptimal actions.

So there are cases where adding noise or perturbation to your demonstration set can lead to a more robust imitation policy[1].

[1] T-Rex https://arxiv.org/abs/1904.06387

Imitation Learning in PRIMAL

As I understand it, the paper you link to only uses expert demonstrations during imitation learning. The suboptimal demonstrations are RL samples being used for an RL objective.

The authors describe their choice to combine IL and RL as follows:

combining RL and IL has been shown to lead to faster, more stable training as well as higher-quality solutions in robot manipulation. These advantages are likely due to the fact that IL can help to quickly identify high-quality regions of the agents state-action space, while RL can further improve the policy by freely exploring these regions