# How to take actions at each episode and within each step of the episode in deep Q learning?

In deep Q learning, we execute the algorithm for each episode, and for each step within an episode, we take an action and record a reward.

I have a situation where my action is 2-tuple $$a=(a_1,a_2)$$. Say, in episode $$i$$, I have to take the first half of an action $$a_1$$, then for each step of the episode, I have to take the second half of the action $$a_2$$.

More specifically, say we are in episode $$i$$ and this episode has $$T$$ timesteps. First, I have to take $$a_1(i)$$. (Where $$i$$ is used to reference episode $$i$$.) Then, for each $$t_i\in\{1,2,\ldots,T\}$$, I have to take action $$a_2(t_i)$$. Once I choose $$a_2(t_i)$$, I get an observation and a reward for the global action $$(a_1(i), a_2(t_i))$$.

Is it possible to apply deep Q learning? If so, how? Should I apply the $$\epsilon$$-greedy twice?

• Do you only wait one timestep after taking action $a_1$ before action $a_2$ is executed? Do you have to choose $a_1$ and $a_2$ simultaneously, or are you given an observation of the environment after executing $a_1$ and then allowed to choose $a_2$? – DeepQZero Jun 5 at 21:11
• At episode $i$, I choose $a_1(i)$. Now, for each time step $t_i$ in episode $i$, I choose $a_2(t_i)$. Only after choosing action $a_2(t_i)$ I get an observation and receive a reward for my chosen action $(a_1(i), a_2(t_i))$ at $(i, t_i)$. – zdm Jun 5 at 21:16
• Just to be sure, after choosing action $a = (a_1, a_2)$, then episode terminates (i.e. you don't choose another action for the remainder of the episode)? – DeepQZero Jun 5 at 21:21
• Yes, $t$ and $t'$ are in the same episode. I am editing the question. – zdm Jun 5 at 21:40
• Can you explain your MDP further? Once you’ve chosen ($a_1,a_2$) do you then have to choose another two tuple before getting the next state and reward? – David Ireland Jun 5 at 21:58