In RL (reinforcement learning) or MARL (multi-agent reinforcement learning), we have the usual tuple:
(state, action, transition_probabilities, reward, next_state)
In MORL (multi-objective reinforcement learning), we have two more additions to the tuple, namely, "preferences" and "preference functions".
What are they? What do we do with them? Can someone provide an intuitive example?
In MORL the reward component is a vector rather than a scalar, with an element for each objective. So if we are using a multiobjective version of an algorithm like Q-learning, the Q-values stored for each state-action pair will also be vectors.
Q-learning requires the agent to be able to identify the greedy action in any state (the action expected to lead to the highest long-term return). For scalar rewards this is easy, but for vector values it is more complicated as one vector may be higher for objective 1, while another is higher for objective 2, and so on.
We need a means to order the vector values in terms of how well they meet the user's desired trade-offs between the different objectives. That is the role of the preference function and preferences. The function defines a general operation for either converting the vector values to a scalar value so they can be compared, or for performing some sort of ordering of the vectors (some types of orderings such as lexicographic ordering can't readily be defined in terms of scalarisation). So, for example, our preference function might be a weighted sum of the components of the vector. The preferences specify the parameters of the preference function which define a specific ordering (i.e. based on the needs of the current user). So, in the case of a weighted sum for the preference function, the preferences would be specified in terms of the values of the weights.
The choice of preference function can have implications for the types of solutions which can be found, or for whether additional information needs to be included in the state in order to ensure convergence.
I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a useful introduction to this area)
Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67-113.
