In this page Limitations on horizon length from the Imitation library, the authors recommend that the user sticks to fixed horizon experiments because there could be "information leak" otherwise.
I'm having problems understanding this term, how can the information leak?
Can someone please explain this with an example, or something that might help me get some intuition about the issue?
In the standard Cartpole environment, the episode ends either at 500 timesteps or when the Cartpole falls down. Expert demonstrations show how to keep the Cartpole upright for 500 timesteps.
One degenerate way for Generative Adversarial Imitation Learning or Adversarial Inverse Reinforcement Learning to train an expert policy is to assign a positive reward to every action taken by the generator policy, regardless of whether it mirrors the expert actions. With a positive reward, the RL algorithm that is training the generator policy learns to favor prolonging the episode.
Hence the episode length is a form of side-channel information that makes the imitation learning task easier, and we want to exclude it by choosing environments with fixed episode length.
