I am new to reinforcement learning, but, for a finite horizon application problem, I am considering using the average reward instead of the sum of rewards as the objective. Specifically, there are a total of $T$ maximally possible time steps (e.g., the usage rate of an app in each time-step), in each time-step, the reward may be 0 or 1. The goal is to maximize the daily average usage rate.

Episode length ($T$) is maximally 10. $T$ is the maximum time window the product can observe about a user's behavior of the chosen data. There is an indicator value in the data indicating whether an episode terminates. From the data, it is offline learning, so in each episode, $T$ is given in the data. As long as an episode doesn't terminate, there is a reward of $\{0, 1\}$ in each time-step.

I heard if I use an average reward for the finite horizon, the optimal policy is no longer a stationary policy, and optimal $Q$ function depends on time. I am wondering why this is the case.

I see normally, the objective is defined maximizing

$$\sum_t^T \gamma^t r_t$$

And I am considering two types of average reward definition.

  1. $1/T(\sum^𝑇_{t=0}\gamma^t r_t)$, $T$ varies is in each episode.

  2. $1/(T-t)\sum^T_{i=t-1}\gamma^i r_i$

  • $\begingroup$ How does your horizon relate to episode length (or is the target environment continuous)? What controls value of $T$? Is the value of $T$ known at $t=0$? What do you mean by "iteration" - is that {0, 1} reward per time step, or is it {0, 1} reward per $t=0$ to $T$ pseudo-episode? $\endgroup$ Aug 10 '20 at 9:58
  • $\begingroup$ @NeilSlater i added the edits $\endgroup$
    – lll
    Aug 10 '20 at 18:55
  • $\begingroup$ Thanks for the edit. So, to be clear, $T$ is not actually the length of an "episode", but the size of an observation window into a continuous process? There is essentially no natural episode, just observations, and some of them form into small trajectories where state progresses due to actions taken? $\endgroup$ Aug 10 '20 at 18:58
  • $\begingroup$ T can be thought of as episode since after maximum of 10 time steps, users will go to another category of the app, which is not what we care about. And yes, it can thought of as no natural episode; state progresses due to actions taken $\endgroup$
    – lll
    Aug 10 '20 at 20:44
  • $\begingroup$ For "I heard if I use an average reward for the finite horizon" I would be interested to know where you heard/read this from, as on the surface it looks incorrect to me for your problem (i.e. I agree with you). There may be some context to it that makes it true for some problems and not others, so I would like to see. Do you have a link? $\endgroup$ Aug 11 '20 at 10:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.