Most RL algorithms assume a discretization of time (although RL can also be applied to continuous-time problems ), i.e., in theory, it doesn't really matter what the actual time between consecutive time steps is, but, in practice, you may have delays in the rewards or observations, so you cannot perform e.g. the TD updates immediately. One natural solution to your problem would be to keep track (e.g. in a buffer) of the reward obtained and the next state that the agent ended up in after having taken a certain action in a certain state, or use some kind of synchronization mechanism (note that I've just come up with these solutions, so I don't know if this has been done or not to solve problems). In practice, this may not work (in all cases), for example, during real-time inference, where you need to decide quickly what you need to do even without full information about the current state or reward.
Note that, in RL, rewards are often said to be delayed, in the sense that
- you may know the consequences of an action only many time-steps after you have taken it (determining the consequences of an action is known as the credit assignment problem), or
- you may get a non-zero reward only when the agent gets to a goal/final state (in this last case, these rewards are also known as sparse).
These two problems are common in RL. However, if I understand correctly your concerns, this is a bit different than your problem, because your problem also involves the potential delay of the state or even reward that was supposed to arrive at a previous time step, which can be due e.g. to an erratic or broken sensor/actuator. For instance, if you are using DQN, which typically builds an approximation of the current state by concatenating the last frames captured by your camera, if you have delays in the frames that cause the natural order of the frames to be changed, this could lead to a bad approximation of the current state, which could actually lead to a catastrophic event. So, yes, this is an important problem that needs to be tackled.
Given that I am not really familiar with the actual existing solutions, I'll refer you to the paper Challenges of Real-World Reinforcement Learning that I read a few weeks ago, which mentions this issue and points you to other research work that attempted to address it. Take a look at this answer too, if you're more interested in delayed/sparse rewards.