# How to deal with the time delay in reinforcement learning?

I have a question regarding the time delay in reinforcement learning (RL).

In the RL, one has state, reward and action. It is usually assumed that (as far as I understand it) when the action is executed on the system, the state changes immediately and that the new state can then be analysed (influencing the reward) to determine the next action. However, what if there is a time delay in this process. For example, when some action is executed at time $$t_1$$, we can only get its effect on the system at $$t_2$$ (You can imagine a flow: the actuator is in the upstream region and the sensor is in the downstream region, so that there will be a time delay between the action and the state). How do we deal with this time delay in RL?

Most RL algorithms assume a discretization of time (although RL can also be applied to continuous-time problems [1]), i.e., in theory, it doesn't really matter what the actual time between consecutive time steps is, but, in practice, you may have delays in the rewards or observations, so you cannot perform e.g. the TD updates immediately. One natural solution to your problem would be to keep track (e.g. in a buffer) of the reward obtained and the next state that the agent ended up in after having taken a certain action in a certain state, or use some kind of synchronization mechanism (note that I've just come up with these solutions, so I don't know if this has been done or not to solve problems). In practice, this may not work (in all cases), for example, during real-time inference, where you need to decide quickly what you need to do even without full information about the current state or reward.

Note that, in RL, rewards are often said to be delayed, in the sense that

1. you may know the consequences of an action only many time-steps after you have taken it (determining the consequences of an action is known as the credit assignment problem), or
2. you may get a non-zero reward only when the agent gets to a goal/final state (in this last case, these rewards are also known as sparse).

These two problems are common in RL. However, if I understand correctly your concerns, this is a bit different than your problem, because your problem also involves the potential delay of the state or even reward that was supposed to arrive at a previous time step, which can be due e.g. to an erratic or broken sensor/actuator. For instance, if you are using DQN, which typically builds an approximation of the current state by concatenating the last frames captured by your camera, if you have delays in the frames that cause the natural order of the frames to be changed, this could lead to a bad approximation of the current state, which could actually lead to a catastrophic event. So, yes, this is an important problem that needs to be tackled.

Given that I am not really familiar with the actual existing solutions, I'll refer you to the paper Challenges of Real-World Reinforcement Learning that I read a few weeks ago, which mentions this issue and points you to other research work that attempted to address it. Take a look at this answer too, if you're more interested in delayed/sparse rewards.