As the question states, I am wondering how, if at all, a varying length of a trajectory (series of state,action pairs) will impact training/performance of policy gradient algorithms such as PPO, TRPO and VPG.
Let's say an agent runs in an environment where the length of each episode may not always be the same (this is true especially in games such as poker). The cumulative reward for longer trajectories will inevitably be larger than the rewards for smaller trajectories. To me, this seems to cause an imbalance favoring the actions of a policy executed for a longer period of time (non terminal state) even if that policy may be sub-optimal compared to the policy applied for a shorter trajectory.
Are my assumptions correct? How does trajectory size end up impacting the updates on a policy?