Some IL approaches train the agents by using some specific ratio of expert demonstrations to trajectories generated using the policy being optimized.
In the specific paper I'm reading they say "we experimented with various IL proportions (10-50% by increments of 10%) and observed that the RL/IL ratio does not seem to affect the performance of the trained policy by much."
My question is: why not rely only on expert demonstrations instead of introducing the noise of trajectories generated by a sub-optimal policy?.
My assumptions are:
- This noise helps explore the state space beyond the specific episodes experienced by the expert system.
- You might have expert systems that only work on a limited state space and do not scale to bigger, more complex environments. Therefore, although you can't use expert demonstrations in the bigger envs you still want to leverage their experience by learning their behavior in limited settings and the way to avoid overfitting to a specific, constrained policy is by always having some proportion of episodes generated by your policy being learned.