1
$\begingroup$

I've been reading about reinforcement learning, and it seems to me that reinforcement learning assumes the environment is static, and therefore that the reward for taking a particular action will be the same from time to time. But what if something happens outside the agent's control, that changes the reward?

As a simplified example, suppose the agent in a RL algorithm is a grape farmer living on a planet known for extreme weather events. Its goal is to maximize profit/minimize losses. Grape seeds cost 1000 dollars. It takes 1 year for the grape seeds to grow delicious grapes. There is a 20% chance of the weather destroying the grapes during the year, leading to the agent losing 1000 dollars. However, if the weather does NOT destroy the grapes, the agent wins $2000.

The agent starts the year with two choices:

  • Grow grapes
  • Don't grow grapes

In its first year, an adverse weather event occurs, killing the grapes. Will the agent ever grow grapes again, or will it forever decide that the best way to win is to not plant grapes? Likewise, if it ever DOES grow grapes, can we trust it to know when not to grow grapes?

$\endgroup$
1
  • $\begingroup$ Why do you say the weather is external to the environment? Your environment is the weather. $\endgroup$ Commented Jan 13, 2022 at 19:57

1 Answer 1

1
$\begingroup$

In the question, you are not describing the environment changing. Instead, there is a fixed 20% chance of a bad weather event each year. Such events can me modelled as a static environment with stochastic results.*

If nothing else happened in the year, it is easy to calculate the expected immediate reward for each action:

Not planting seeds. $0$

Planting seeds. $-1000 + 0.8 \times 2000 = 600$

So the question resolves to: Can a learning algorithm discover these odds, and make the correct decision, based only on its experience of individual years? Perhaps it will be unlucky in its first year and simply give up?

The answer is yes, most reinforcement learning (RL) algorithms will cope just fine, and learn that planting seeds is the best option in the simple environment that you have proposed.

There is an important caveat: To learn the true value of each option, algorithm must be one that explores. That means, whichever action the agent thinks is currently best, there must be some chance that it decides to take (what it thinks is) a worse option. If the worse option turns out better than it expected, the agent can adjust its expectations, and maybe choose that one more frequently in future, and vice versa if the option turns out worse than expected.

Nearly all RL methods do this, by setting some kind of exploration parameter. The more the agent explores, the more it will learn about alternatives to its current best guess at optimal behaviour. However, whilst exploring it may pay the price for making many sub-optimal decisions. Your planning agent may decide to not plant anything every few years, just to see if that is better, even after it has established that planting is the best way to get profit.

It is often hard to balance between the learning obtained by trying different options vs doing what the agent currently thinks is best. This is called the exploration-exploitation tradeoff, and is a fundamental issue in RL.


* For comparison, a non-stationary environment in RL would be one in which the probabilities and/or reward distributions change so that values learned from older experience are no longer valid. If climate change impacted your grape-growing region such that after some year the chance of a bad storm was 80%, and the agent had no way of knowing that in advance, then the agent might lose money trying to follow its previously best policy. If it was still learning, it would adapt to these new rules eventually though.

$\endgroup$
2
  • $\begingroup$ Ok, so suppose the weather changes not by chance, but by motivation. There is some superhuman or other entity that might want to alter the weather such that the grapes die. Can the agent learn to adapt? $\endgroup$ Commented Jan 14, 2022 at 2:11
  • $\begingroup$ @MontanaBurr: That depends on what you mean by "motivation". You need to be clear in your definitions if you want an answer. Most importantly, is the "motivation" based on some non-random factor, and can the agent observe or predict that factor? This is quite different from your original question, needs clarification, and there could be lots of detail in any answer, so I suggest you ask a new question about that. $\endgroup$ Commented Jan 14, 2022 at 7:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .