7
$\begingroup$

How do you actually decide what reward value to give for each action in a given state for an environment?

Is this purely experimental and down to the programmer of the environment? So, is it a heuristic approach of simply trying different reward values and see how the learning process shapes up?

Of course, I understand that the reward values have to make sense, and not just put completely random values, i.e. if the agent makes mistakes then deduct points, etc.

So, am I right in saying it's just about trying different reward values for actions encoded in the environment and see how it affects the learning?

$\endgroup$
1
  • $\begingroup$ Here is a related question. $\endgroup$
    – nbro
    Oct 7 '20 at 17:01
4
$\begingroup$

In Reinforcement Learning (RL), a reward function is part of the problem definition and should:

  • Be based primarily on the goals of the agent.

  • Take into account any combination of starting state $s$, action taken $a$, resulting state $s'$ and/or a random amount (a constant amount is just a random amount with a fixed value having probability 1). You should not use other data than those four things, but you also do not have to use any of them. This is important, as using any other data stops your environment from being a Markov Decision Process (MDP).

  • Given the first point, as direct and simple as possible. In many situations this is all you need. A reward of +1 for winning a game, 0 for a draw and -1 for losing is enough to fully define the goals of most 2-player games.

  • In general, have positive rewards for things you want the agent to achieve or repeat, and negative rewards for things you want the agent to avoid or minimise doing. It is common for instance to have a fixed reward of -1 per time step when the objective is to complete a task as fast as possible.

  • In general, reward 0 for anything which is not directly related to the goals. This allows the agent to learn for itself whether a trajectory that uses particular states/actions or time resources is worthwhile or not.

  • Be scaled for convenience. Scaling all rewards by a common factor does not matter at a theoretical level, as the exact same behaviour will be optimal. In practice you want the sums of reward to be easy to assess by yourself as you analyse results of learning, and you also want whatever technical solution you implement to be able to cope with the range of values. Simple numbers such as +1/-1 achieve that for basic rewards.

Ideally, you should avoid using heuristic functions that reward an agent for interim goals or results, as that inserts your opinion about how the problem should be solved into the system, and may not in fact be optimal given the goals. In fact you can view the purpose of value-based RL is learning a good heuristic function (the value function) from the more sparse reward function. If you already had a good heuristic function then you may not need RL at all.

You may need to compare very different parts of the outcome in a single reward function. This can be hard to balance correctly, as the reward function is a single scalar value and you have to define what it means to balance between different objectives within a single scenario. If you do have very different metrics that you want to combine then you need to think harder about what that means:

  • Where possible, flatten the reward signal into the same units and base your goals around them. For instance, in business and production processes if may be possible to use currency as the units of reward and convert things such as energy used, transport distance, etc., into that currency.

  • For highly negative/unwanted outcomes, instead of assigning a negative reward, consider whether a constraint on the environment is more appropriate.

You may have opinions about valid solutions to the environment that you want the agent to use. In which case you can extend or modify the system of rewards to reflect that - e.g. provide a reward for achieving some interim sub-goal, even if it is not directly a result that you care about. This is called reward shaping, and can help in practical ways in difficult problems, but you have to take extra care not to break things.

There are also more sophisticated approaches that use multiple value schemes or no externally applied ones, such as hierarchical reinforcement learning or intrinsic rewards. These may be necessary to address more complex "real life" environments, but are still subject of active research. So bear in mind that all the above advice describes the current mainstream of RL, and there are more options the deeper you research the topic.

Is this purely experimental and down to the programmer of the environment. So, is it a heuristic approach of simply trying different reward values and see how the learning process shapes up?

Generally no. You should base the reward function on the analysis of the problem and your learning goals. And this should be done at the start, before experimenting with hyper-parameters which define the learning process.

If you are trying different values, especially different relative values between different aspects of a problem, then you may be changing what it means for the agent to behave optimally. That might be what you want to do, because you are looking at how you want to frame the original problem to achieve a specific behaviour.

However, outside of inverse reinforcement learning, it is more usual to want an optimal solution to a well-defined problem, as opposed to a solution that matches some other observation that you are willing to change the problem definition to meet.

So, am I right in saying it's just about trying different reward values for actions encoded in the environment and see how it affects the learning?

This is usually not the case.

Instead, think about how you want to define the goals of the agent. Write reward functions that encapsulate those goals. Then focus on changes to the agent that allow it to better learn how to achieve those goals.

Now, you can do it the way round, as you suggest. But what you are doing, in that case, is changing the problem definition, and seeing how well a certain kind of agent can cope with solving each kind of problem.

$\endgroup$
3
  • $\begingroup$ Given that this is a long answer (but I would say good), I would suggest that you edit it to make some of the most important keywords bold (but, of course, this is not strictly necessary and it's just my opinion to improve the readability). Anyway, you say "In fact you can view the purpose of value-based RL is learning a good heuristic function (the value function) from the more sparse reward function. If you already had a good heuristic function then you may not need RL at all.", but I think this can be misleading because one could think that RL is typically applied when the RF is sparse. $\endgroup$
    – nbro
    Oct 7 '20 at 17:15
  • $\begingroup$ However, this may not be the case, and sparse reward functions (that lead to slow learning) is actually a research problem, i.e. if the reward signal was richer/denser (and still "correct"), that would probably speed up learning. I am also not sure why you say we wouldn't need RL. Maybe you're talking about the case where you have "labels", so this would turn into a SL problem? $\endgroup$
    – nbro
    Oct 7 '20 at 17:16
  • 1
    $\begingroup$ @nbro: I think the core problem is that I mixed different meanings of "heuristic" in that paragraph. A true heuristic in the sense I use at the end would look a lot like an optimal value function, but I also used the term to mean "helpful additional rewards", which is different. I should have used two different terms to make it clearer. I'll have a think about a suitable edit $\endgroup$ Oct 7 '20 at 17:53
2
$\begingroup$

Yes, you are right. It is somehow an arbitrary choice, although you should consider the reasonable numerical ranges of your activation functions if you decide to go beyond the values +/- 1.

You can also have a think about whether you want to add a small reward for the agent reaching states that are near the goal, if you have an environment where such states are discernable.

If you want to have a more machine learning approach to reward values, consider using an Actor-Critic arrangement, in which a second network learns reward values for non-goal states (by observing the results of agent exploration), although you still need to determine end-state values according to your handcrafted heuristic.

$\endgroup$
1
  • $\begingroup$ You say "although you should consider the reasonable numerical ranges of your activation functions", but RL may not use neural networks (but of course that's possible). Maybe you meant "reward functions" rather than "activation functions"? $\endgroup$
    – nbro
    Oct 7 '20 at 17:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.