In the original paper, the objective of PPO is as follows:enter image description here. My question is, how does this objective behave in a sparse reward setting (i.e., reward is only given after a sequence of actions were taken)? In this case we don't have $\hat{A}_{t}$ defined for every $t$.

  • $\begingroup$ Why won't we A_{hat}_{t} for every t? There is usually a tuple of (state, reward, next_state) associated with every time step t. For sparse reward settings, the reward will be 0 for non-reward states. $\endgroup$ Mar 6, 2023 at 1:40
  • $\begingroup$ @desert_ranger yes you can think of reward as 0 for those states, but in some situations it might be undefined $\endgroup$
    – Sam
    Mar 6, 2023 at 10:58
  • $\begingroup$ It is the user who designs the reward for each step. Therefore, as long as the environment is formulated correctly, this shouldn't happen. $\endgroup$ Mar 6, 2023 at 23:28
  • $\begingroup$ @desert_ranger think of Go. By default, not every move has a reward assigned. Are you suggesting going down the reward-shaping route to introduce artificial rewards? $\endgroup$
    – Sam
    Mar 7, 2023 at 2:45
  • 1
    $\begingroup$ @DavidIreland yeah so to fit into MDP framework, can assign dummy reward 0 to the intermediate states. $\endgroup$
    – Sam
    Apr 6, 2023 at 14:06


You must log in to answer this question.

Browse other questions tagged .