# How does Q-learning work in stochastic environments?

The Q function uses the (current and future) states to determine the action that gets the highest reward.

However, in a stochastic environment, the current action (at the current state) does not determine the next state.

How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?

• Could you clarify: "In a stochastic environment, where the action doesn't influence state" - do you mean that the state is not 100% predictable, and the action doesn't fully determine the state? E.g. deciding to move forward, and a game then moves you 1d6 steps forward. Or do you really mean that the action choice has absolutely no influence on the resulting state - and the state will evolve by rules independent of the action choice? E.g. deciding to buy or sell, getting some reward, then the state just changes randomly. The answer you need is very different depending on which case you mean. Mar 31, 2018 at 12:20
• @NeilSlater, Thanks for your response. In this case the action has no influence at all on the next state. The state will evolve by rules independent of the action choice. Apr 2, 2018 at 19:00
• In which case, although Q learning should work without errors, you would be expending resources for the agent to learn the fact that the action does not influence state. Your situation is more commonly known as "contextual bandit" and there a variety of solvers out there. The positive news is that good old supervised learning should be fine to learn from the historical data. Then it's just a matter of how online your algorithm needs to be, and how risk averse whilst making online decisions. Apr 2, 2018 at 19:11

How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?

The learned $$Q$$-function is not only used during training, but also after training (in what we may call "deployment", when we expect a trained agent to behave according to what it has learned).

However, the reliance on future states is only there during training, it is no longer required for deployment.

During training, we use the following $$Q$$-learning update rule:

$$Q(s, a) \gets (1 - \alpha) Q(s, a) + \alpha \left( R + \gamma \color{red}{\max_{a'} Q(s', a')} \right),$$

where $$s'$$ is the state we reach after executing $$a$$ in $$s$$. Here, the $$\color{red}{\text{red}}$$ part is the part where we rely on knowledge of the future $$s'$$. This is available in training because we can simply pick $$a$$, execute it in $$s$$, observe $$s'$$, and only then trigger our update step.

Outside of training (and actually also during training), we also rely on our $$Q(s, \cdot)$$ function for the selection of actions. We typically select an action $$a$$ according to $$a = \arg\max_a Q(s, a)$$; we select the action $$a$$ that maximises $$Q(s, a)$$ in our current state $$s$$. The important thing to note here is that there is no $$s'$$ term in this description of how we select actions: we do not require knowledge of our future state.

Note: in my answer I decided to answer the question literally as it is written, i.e. I'm explaining how $$Q$$-learning can still work in the described setting where actions $$a$$ have no influence whatsoever on the future state reached.

In practice, I would never recommend actually using $$Q$$-learning in such a setting, and instead refer to Neil Slater's comment about Contextual Multi-Armed Bandit algorithms likely providing a better solution.