0
$\begingroup$

I found this article quite useful on how to shape a reward function in RL. However, the example they gave is quite simple, where the goal is to minimize only two quantities (velocity and distance).

How would you formulate the reward function if you had, for instance, 4 quantities to optimize?

enter image description here

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ At AI Stack Exchange we prefer questions to be self-contained. This one requires going off site to read the article to even understand the basics of what you are asking. Could you please summarise the article in a couple of short paragraphs - what was the environment, how was the reward function shaped (give the equation, you can use LaTex inside $ like this $r = x_{t} - x_{t+1}$ which becomes $r = x_{t} - x_{t+1}$), and how did it help? $\endgroup$
    – Neil Slater
    Jun 17, 2021 at 8:28
  • $\begingroup$ I mentioned the article just as an example, my question is just that I need to formulate a reward function to optimize 4 parameters in the example that I sent they tried to optimize only 2 parameters as shown in the image that I provided now! Thanks for your reply. $\endgroup$
    – BAKYAC
    Jun 17, 2021 at 9:24
  • $\begingroup$ check out this answer $\endgroup$
    – Aray Karjauv
    Jun 17, 2021 at 12:14
  • $\begingroup$ @BAKYAC Could you please at least provide more details about your concrete problem? You say that you have 4 quantities to optimize. What quantities/variables are these? What problem are you trying to solve with RL? What is the action and state spaces? $\endgroup$
    – nbro
    Jun 17, 2021 at 23:51

1 Answer 1

Reset to default
1
$\begingroup$

Here is how I managed to construct a reward function in one of my projects, where I trained an RL model for a self-driving robot that has only a single camera to navigate through a tunnel:

$$ R = \left\{ \begin{array}{ll} d_m - 3 - \left| d_l - d_r \right| & \text{if not terminal state} \\ -100 & \text{otherwise} \end{array} \right. $$

where $d_m$ is the middle distance, $d_l$ is the left distance and $d_r$ is the right distance, and $d_m, d_l, d_r \in [0, 10]$. To get this information, the agent has a laser sensor. The agent does not directly observe these distances. Instead, it gets a real number as the reward signal that indicates how good the agent is performing an action and tries to map the camera view to it. This function is designed so that the agent should stay in the middle of the tunnel, $-\left|d_l - d_r\right|$, and has to avoid head-on collisions, $d_m - 3$. Thus, the highest possible reward is $R = 10 - 3 - \left| 10 - 10 \right| = 7$

enter image description here

Basically, you can use any number of parameters in your reward function as long as it accurately reflects the goal the agent needs to achieve. For instance, I could penalize the agent for frequent steering on straight sections so that it drives smoothly.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Worth adding the need to weight any separate reward calculations appropriately before combining. When you have multiple possibly conflicting objectives, it is down to you as system designer to set the rules for which is more important, the RL agent cannot balance them for you $\endgroup$
    – Neil Slater
    Jun 17, 2021 at 15:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .