2
$\begingroup$

I have a basic question. I'm working towards developing a reward function for my DQN. I'd like to train an RL agent to edit pixels on an image. I understand that convolutions are ideal for working with images, but I'd like to observe the agent doing it in real-time. Just a fun side project.

Anyway, to encourage an RL agent to craft a specific image I'm crafting a reward function that returns a $N \times N$ dimensional matrix. Which represents the distance between the state of the target image (RGB values for each pixel location) and the image the agent crafted.

Generally speaking, is it better for rewards to be a scalar, or is using matrices okay?

Improve this question
$\endgroup$
2
  • $\begingroup$ How would you adapt DQN to work with vectorized rewards? Note that I changed the title to be what I think is your actual question. If that's not your actual question, let us know what it is. $\endgroup$
    – nbro
    Feb 2, 2021 at 16:00
  • $\begingroup$ See also this and this. $\endgroup$
    – nbro
    Feb 2, 2021 at 22:28

1 Answer 1

Reset to default
3
$\begingroup$

Generally speaking, is it better for rewards to be a scalar, or is using matrices okay?

Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and reinforcement learning (RL) methods.

Although it is possible to accumulate matrices in various ways, by e.g. simple matrix addition, and come up with an analog for expected return which would be a weighted sum of matrices, you then get stuck. There is no fixed way to rank matrices and decide whether one is a better result than another. This is a requirement for any learning process that aims to improve at a task - it needs feedback that changes it makes are better or worse related to some reference. As a result, most objective functions and metrics in optimisation use real-valued scalars, which can always be placed into order to decide a highest or lowest value.

This does not prevent you using a matrix representation for your project, if it is a natural fit. To turn it into a usable reward, you will need to convert that matrix into a real-valued metric. Perhaps the L2 norm or other standard measure that summarises the matrix will be good for your task.

It is possible to process multiple scalar rewards at once with single learner, using multi-objective reinforcement learning. Applied to your problem, this would give you access to a matrix of policies, each of which maximised the reward value of one cell within the matrix. It also allows for switching between objectives in a hierarchical manner using a "policy of policies" if some preference for what to achieve changes. It is not 100% clear, but I do not think this is what you want to do.

Improve this answer
$\endgroup$
2
  • $\begingroup$ It might be worthing explicitly mentioning in your answer "multi-objective reinforcement learning" and the "reward hypothesis". This survey also seems a good reference for further details on MORL (i.e. it formulates the MDPs, value functions, etc., in the case of vector-valued rewards). $\endgroup$
    – nbro
    Feb 2, 2021 at 22:35
  • $\begingroup$ @nbro: Thanks, I'll have athink about that. In this case MORL is a bit of a red herring as OP appears to have a singular task. If I have it right, the vector of rewards in MORL represents multiple separate goals to be processed at once, such that a higher order goal selector policy can be used in a hierarchical fashion. It is not a way to avoid needing scalar reward values for optimality, but a way to parallel process multiple scalar reward schemes in an otherwise identical environment. $\endgroup$
    – Neil Slater
    Feb 3, 2021 at 7:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .