In deep Q learning, we execute the algorithm for each episode, and for each step within an episode, we take an action and record a reward.

I have a situation where my action is 2-tuple $a=(a_1,a_2)$. Say, in episode $i$, I have to take the first half of an action $a_1$, then for each step of the episode, I have to take the second half of the action $a_2$.

More specifically, say we are in episode $i$ and this episode has $T$ timesteps. First, I have to take $a_1(i)$. (Where $i$ is used to reference episode $i$.) Then, for each $t_i\in\{1,2,\ldots,T\}$, I have to take action $a_2(t_i)$. Once I choose $a_2(t_i)$, I get an observation and a reward for the global action $(a_1(i), a_2(t_i))$.

Is it possible to apply deep Q learning? If so, how? Should I apply the $\epsilon$-greedy twice?

  • $\begingroup$ Do you only wait one timestep after taking action $a_1$ before action $a_2$ is executed? Do you have to choose $a_1$ and $a_2$ simultaneously, or are you given an observation of the environment after executing $a_1$ and then allowed to choose $a_2$? $\endgroup$ – DeepQZero Jun 5 at 21:11
  • $\begingroup$ At episode $i$, I choose $a_1(i)$. Now, for each time step $t_i$ in episode $i$, I choose $a_2(t_i)$. Only after choosing action $a_2(t_i)$ I get an observation and receive a reward for my chosen action $(a_1(i), a_2(t_i))$ at $(i, t_i)$. $\endgroup$ – zdm Jun 5 at 21:16
  • $\begingroup$ Just to be sure, after choosing action $a = (a_1, a_2)$, then episode terminates (i.e. you don't choose another action for the remainder of the episode)? $\endgroup$ – DeepQZero Jun 5 at 21:21
  • 1
    $\begingroup$ Yes, $t$ and $t'$ are in the same episode. I am editing the question. $\endgroup$ – zdm Jun 5 at 21:40
  • 1
    $\begingroup$ Can you explain your MDP further? Once you’ve chosen ($a_1,a_2$) do you then have to choose another two tuple before getting the next state and reward? $\endgroup$ – David Ireland Jun 5 at 21:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.