I'm training a DQN in a real environment where I do not have a natural terminal state, so I've built the episode in an artificial way (i.e. it starts in a random condition and after T steps it ends). My question is about the terminal state: should I consider it when I have to compute $y$ (so using only the reward) or not?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ I think you can use Deep Q learning with the average reward for a non-episodic task, so you don't need to terminate the environment artificially. $\endgroup$– Swakshar DebAug 19, 2020 at 7:11
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
If the episode does not terminate naturally, then if you are breaking it up into pseudo-episodes for training purposes, the one thing you should not do is use the TD target $G_{T-1} = R_T$ used for an end of episode, which assumes a return of 0 from any terminal state $S_{T}$. Of course that is because it is not the end of the episode.
You have two "natural" options to tweak DQN to match to theory at the end of a pseudo-episode:
Store the state, action, reward, next_state tuple as normal and use the standard one step TD target $G_{t:t+1} = R_{t+1} + \gamma \text{max}_{a'} Q(S_{t+1}, a')$
Completely ignore the last step and don't store it in memory. There is no benefit to this as opposed to the above option, but it might be simpler to implement if you are using a pre-built RL library.
Both these involve ignoring any done flag returned by the environment for the purposes of calculating TD targets. You still can use that flag to trigger the end of a loop and a reset to new starting state.
You should also take this approach if you terminate an episodic problem early after hitting a time step limit, in order to reset for training purposes.
As an aside (and mentioned in comment by Swakshar Deb), you can also look into the average reward setting for non-episodic environments. This solves the problem of needing to pick a value for $\gamma$. If you have no reason to pick a specific $\gamma$ in a continuing problem, then it is common to pick a value close to 1 such as 0.99 or 0.999 in DQN - this is basically an approximation to average reward.
answered Aug 19, 2020 at 8:20
-
$\begingroup$ So can I use Deep Q-Learning with the differential reward for a non-episodic task with γ=1? Why if γ=0.99 as you said then this approximates the differential reward? $\endgroup$ Aug 19, 2020 at 9:03
-
1$\begingroup$ @ddaedalus: If you use differential reward you should not use $\gamma$ at all. Mathematically $\gamma = 1$ is the same, but better to think of average reward as not having any discount factor at all. When $\gamma \sim 1$ in a discounted return, then the measure is very similar to average reward in expectation, just different by a scale factor. So a policy that maximises expected discounted return with a high $\gamma$ and no terminal states will also be very similar to a policy that maximises average reward. $\endgroup$ Aug 19, 2020 at 9:32