I read this article: "Towards Autonomous Data Ferry Route Design through Reinforcement Learning" by Daniel Henkel and Timothy X Brown. It specifies an infinite horizon problem where they use as a reward function for TD[0] the following:

\begin{equation} r(s,a) = - \int_{t_0}^{t_1} (Ft +N_0)e^{-\beta t} dt \end{equation}

where $N_0$ and $F$ are constant, $\beta$ is used to adjust the discount factor and $t_0 , t_1$ are the initial and final time.

Then they proceed to use $e^{-\beta t}$ as the $\gamma$ (discount factor) in the TD[0] update formula and in the policy formula.

Why is the discount factor in the infinite horizon problem $e^{-\beta t}$, and why is it used as $\gamma$ in $V(s)$ update, since it is a variable factor?

Also, in the formula of the TD[0] update they don't subtract $\alpha V(s)$. They do: \begin{equation} V_{t+1}(s) = V_t(s) + \alpha( r(s,a) + e^{-\beta t_a} V_t(s')) \end{equation} I really think this is a mistake, and the values of $V$ will explode without it, even in an infinite horizon problem. Am I correct ? Is $- V(s)$ missing inside the brackets?

Finally, if someone is willing and has the time to directly read this part of the article and enlighten me on if $t_0$ and $t_1$ represent the initial and final time of an action OR $t_0$ is always 0 and $t_1$ is the duration of the action, I would appreciate it. I ask this because from what is written in the paper $t_0$ seems to be the current time in the simulation, but I'm afraid that would just decay too fast and after some actions the reward would be close to 0. It is really not well explained and I'm a bit confused.

Thank you for your time if you got this far reading. Any guideline answer will be very much appreciated.

  • 1
    $\begingroup$ You should link the article, or at least give the authors and publication, if you hope for someone to read it or refer to it in their answer. Otherwise in order to help you, someone will need to go search for the article. Use edit to add details. $\endgroup$ – Neil Slater Jun 1 at 8:21
  • 1
    $\begingroup$ Hi Miguel. Please, see this ai.meta.stackexchange.com/q/1527/2444. $\endgroup$ – nbro Jun 1 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.