What's the right way of building a deep Q-network?

Question

I'm new to RL and to deep q-learning and I have a simple question about the architecture of the neural network to use in an environment with a continous state space a discrete action space.

I tought that the action $a_t$ should have been included as an input of the neural network, togheter with the state. It also made sense to me as when you have to compute the argmax or the max w.r.t. $a_t$ it was like a "standard" function. Then I've seen some examples of networks that had as inputs only $s_t$ and that had as many outputs as the number of possible actions. I quite understand the logic behind this (replicate the q-values pairs of action-state) but is it really the correct way? If so, how do you compute the $argmax$ or the $max$? Do I have to associate to each output an action?

Answer 1

Do I have to associate to each output an action?

You are absolutely correct! In DQN, each output node of the neural network will be associated with one of your possible actions (assuming a finite discrete action space). After passing an input through the network, the value of each output node is the estimated q-value of the corresponding action. One benefit of this architecture is that you only need to pass the input through the neural network once to compute the q-value of each action, which is constant in the number of actions. If you were to include an action as an input to the neural network along with an observation, then you would need to pass an input for each action, which scales linearly in the number of actions. This is mentioned in paragraph 2 of Section 4.1 in the original DQN paper (https://arxiv.org/abs/1312.5602)

Is it really the correct way? If so, how do you compute the argmax or the max?

It's one possible way that is used in many popular algorithms such as DQN. To find the argmax, you simply take the action corresponding to the output node with highest q-value after passing an input through the network.