What is the name of the reward function that utilizes the rewards of the next n steps?

Question

I have a problem with continuous time, observation and action space. I am discretizing the time to be able to apply the usual Reinforcement Learning algorithms (I chose PPO). The problem consists of a simulated car that has to drive through a series of goals without collisions.

I read chapter 7 of the Reinforcement Learning Book by Sutton and Barto: http://incompleteideas.net/book/the-book-2nd.html

In chapter 7 they describe n-step bootstrapping which uses the rewards of the next n-steps and a value estimate of the nth step to update the value of the current step t. This describes extensions to reinforcement learning algorithms, I would like to use an approach inspired by the n-step bootstrapping.

I would like to use a related approach to smooth my reward function. Unfortunately I am not sure how that approach is called and if it has already been talked about in research.

I am searching a name or some research regarding the following reward function:

Reward(st, at) is simply the reward encountered at timestep t. I would then hand the SmoothedReward to the RL algorithm (PPO).

Can you please direct me to research about something like my smoothed reward function?

Answer 1

Your function could be called the truncated return - i.e. the sum of rewards up to some time step in the future.

It would be unusual to perform reward shaping by taking the orginal reward from an environment and replacing it with the truncated return. It definitely breaks some things, so is not a general solution. The broken things include:

• Off-policy value estimations. The truncated returns would include results from future decisions that the behavior policy made, but the target policy would not make. By "squashing" the return into the new observed reward, you would prevent use of any approach, such as importance sampling, to correct for this mismatch.
• An environment with important rewards before step $n$. These early rewards will be under-represented in the new full returns (i.e. returns calculated on your transformed environment), and this could change which early actions were optimal, changing the environment so that the new agent that is optimal in your reward-shaped environment would make different decisions to the optimal agent in the original environment.

In comments you would use the calculated truncated return from the original environment as the reward signal observed by the PPO algorithm (effectively in a transformed or reward-shaped version of the environment). This is not necessary, because PPO already calculates and uses estimated returns from the observed rewards. This is true for all RL algorithms - they all take the observed return, or calculate an approximate expected return, or sometimes a value derived from the return, such as the advantage function. PPO uses a variation of the advantage function.

Effectively, other that the above two problem cases above, what you would be doing for most on-policy algorithms is multiplying each reward by a factor $\sum_{k=0}^n \gamma^k$, because the same immediate reward will be associated with every state, action pair that came before it, going back n steps. So, this should still work, but I don't think it will transform a problem to make it easier to solve for the agent. It is not much different, if you avoid the problem scenarios, from changing the learning rate.

Some RL agents do use the truncated return internally. It can be used for eligibility traces for example.