One of Reinforcement Learning's core features is the ability to deal with delayed rewards/punishments - i.e. rewards that may occur as a consequence of a decision that occurred multiple time steps ago. That is because the value that it optimises is defined as a long-term sum of immediate rewards. This value is often called the return or utility and although it will take short term rewards into account, they do not have to be the dominant factors, and often are not.
All RL solvers are designed to solve this problem, it is core to the MDP formalism used in RL theory. You can decide to focus on immediate rewards or longer-term ones by adjusting using a discount factor (often represented as $\gamma$ in RL equations). Low discount factors will cause an agent to prefer short-term rewards, high discount factors will cause the agent to prefer for longer-term sums of reward. In continuous problems you must use $\gamma \lt 1$ or instead use average reward as the objective to be maximised.
One classic example of this kind of delay problem is the Mountain Car environment. Here, reward is only granted for reaching a certain point, and it is not possible to reach it by just taking the obvious action of moving directly towards the objective. Most RL algorithms can solve this problem, and get close to optimal solutions, purely from experiencing the final reward and associating it progressively with earlier and earlier decisions.
In practical systems how is this "lag" accounted for?
As above, this is inherent to RL, as opposed to simplified versions such as Contextual Bandits which don't have time steps and evolving state.
In practice, it may take some experimentation to find the best RL algorithm and solve a problem efficiently. Environments with very large delays and lots of noise are harder to solve.
It is also really important to set the reward mechanism for your true goals, and not some intermediate ones. If your goal is a stable state, you need to reward that, and not necessarily maximum throughput. If both throughput and stability are important to the end result, you may need to reward both - e.g. reward throughput, but penalise short-term fluctuations.
How are the un-settled (false) results ignored?
In general handling short-term vs long term effects of actions is known as the assignment problem.
This is most often solved statistically, over long term experience, as the agent associates not just immediate reward with an action, but the long term return after making each decision. Different RL solvers have different mechanisms for effectively moving this association back through time.
The key thing here is that an RL solver learns to maximise expected return or utility from any given state. Working with the expected value allows for smoothing out of variations, provided the agent gains enough experience to approach a statistical mean. Extremely rare and large variations can cause problems with this, and might need to be explicitly coded for as opposed to learned.
How is this noise distinguished from exogenous factors (un-controlled system inputs e.g. an open window exposed to wind)?
This is trickier, and depends more critically on the problem.
A RL-based system that is exposed to these variations whilst learning, and some way to represent their effects in the state variables should learn to pick the best action when these external factors impact it.
RL systems based on maintaining a stable state can sometimes do well when they are moved out of that stable state. A good example here might be pole balancing, where the agent learns enough variations of position and speed that if something knocks the pole out of balance, it will still right itself even though the agent was not trained to react to such events directly.
However, in some cases, if knowledge is not available to the agent, or it has not been trained in that particular scenario, it will fail to generalise and may behave in counter-productive ways.
Either way, if you are concerned about specific scenarios, you need to include them in training, or at least test for them after training.
