What is the main difference between additive rewards and discounted rewards?

Question

What is the difference between additive and discounted rewards?

Answer 1

Discounted reward has its opposite undiscounted reward. The difference between these is that discounted has multiplier gamma != 1 and undiscounted is gamma = 1. Gamma tells the multiplier, how much previous values are valued in next iterations. [1]

Additive refers to different thing, a note found in [2]:

An additive reward function decomposes the reward into a number of contributions and can be represented as a (non-partiotioning) MPA.

This short excerpt does not reveal lot, but was the only I could find that made some sense to me. What I could indeed find out was that although they seem similar concepts by name, they appear totally different by nature.

Sources:

[1] https://en.wikipedia.org/wiki/Reinforcement_learning

[2] The Logic of Adaptive Behavior: Knowledge Representation and Algorithms for Adaptive Sequential Decision Making Under Uncertainty In First-Order and Relational Domains.- M. Van

Answer 2

I will disagree slightly with @mico. There is a usage of "additive rewards" that refers to decomposable reward functions (e.g. my reward in selling an item I do not want to own is composed of the reward of not having an unwanted item anymore, plus the monetary gain in selling the item). But, there is indeed a fundamental relation between additive and discounted rewards. Additive rewards are formulated simply as

$$ R([s_0, s_1, s_2, ...]) = R(s_0) + R(s_1) + R(s_2) + \cdots $$

whereas discounted rewards include a discount factor $\gamma \in [0,1]$ such that

$$ R([s_0, s_1, s_2, ...]) = R(s_0) + \gamma R(s_1) + \gamma^2R(s_2) + \cdots $$

Intuitively, the additive reward for a sequence of states is simply the sum of the rewards acquired at each state, while discounted rewards include a multiplicative discount factor that reduces the influence of rewards as time goes on. You will typically see additive rewards for finite-horizon problems, i.e. you have a discrete number of timesteps to optimize over, and discounted rewards are more relevant for infinite-horizon problems, i.e. you may need to optimize over an infinite number of timesteps (or at least a very large number). The discount factor governs the agent’s greediness in achieving immediate reward, where very small discount factors (closer to 0) encourage the agent to only seek rewards in proximal states, and very large discount factors (closer to 1) encourage the agent to think further into the future about what rewards it can expect to achieve in states it will visit later.

The most direct reference I found for this distinction is in these course slides, which are somewhat authoritative since they from Andrew Barto, a co-author of the de facto text on reinforcement learning.