1
$\begingroup$

My question is about whether is it possible to implement delayed reward logic within Gym environment.

More specifically, I work on ride-pooling RL algorithm, when the action (choice of the parameters of ride-pooling heuristic) is taken once in several simulation iterations. However, trips that are influenced by this action may finish later than next Gym step starts. I am wondering if this is possible to send reward back to one of the previous Gym steps when it will eventually be calculated.

For example, we made first action at simulation iteration 10, second action at 20, third action at 30, and only at 35 the reward for the first action is ready.

In basic implementation of Gym step() one needs to return rewards in the end of the step.

$\endgroup$

1 Answer 1

2
$\begingroup$

I think what you have here (with an important caveat, which I will get to later) is a common misunderstanding about how rewards should be structured for a reinforcement learning (RL) problem. It is perfectly OK to return a reward of $0$ for most steps, and only return measured rewards when the event they are related to happens.

The task of figuring out that the critical action that caused a reward was some number of time steps ago is called the credit assignment problem. Sometimes this can make solving an RL problem hard. However, it is a core issue in RL, and all RL algorithms are built to solve this issue. This is usually achieved through algorithms that backup through time iteratively, assigning expected future values to earlier states (in value-based algorithms), or by associating all actions taken with rewards that occured in their future and allowing this to average out over multiple different experiences (in policy gradient methods).

The important caveat is this: When the agent takes the earlier action, this choice must have had some observable influence on the state, and that change should still persist in some form and to some degree at the time the associated reward is observed, such that it is possible for the agent to make the association. In some cases - e.g. a game of checkers - then the state change appears naturally in how the environment is expressed. In other cases you may need to take extra steps. In the worst-case scenario, you would need to store a history of actions taken and add that to the state vector.

Most simple evironments provided with Gym should provide you with a fully usable state vector where all influences on next state and reward are included. So you may need to do nothing other than apply a standard RL control method. Custom, or more advanced environments may require that you add something extra to the state yourself.

$\endgroup$
1
  • $\begingroup$ Exactly, you say it very well. I upvoted your answer $\endgroup$ Dec 5, 2023 at 22:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .