Why does Deep Q Network outputs multiple Q values?

Question

I am learning Deep RL following this tutorial: https://medium.freecodecamp.org/an-introduction-to-deep-q-learning-lets-play-doom-54d02d8017d8

I understand everything but one detail:

This image shows the difference between a classic Q learning table and a DNN. It states that a Q table needs a state-action pair as input and outputs the corresponding Q value whereas a Deep Q network needs the state as feature input and outputs the Q value for each action that can be made in that state.

But shouldn't the state AND the action together be the input to the network and the network just outputs a single Q value?

Answer 1

I think this was just a "clever" design choice. You can actually design a neural network (NN), to represent your Q function, which receives as input the state and an action and outputs the corresponding Q value. However, to obtain $\max_aQ(s', a)$ (which is a term of the update rule of the Q-learning algorithm) you would need a "forward pass" of this network for each possible action from $s'$. By having a NN that outputs the Q value for each possible action from a given $s'$, you will just need one forward pass of the NN to obtain $\max_aQ(s', a)$, that is, you pick the highest Q value among the outputs of your NN.

In the paper A Brief Survey of Deep Reinforcement Learning (by Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage and Anil Anthony Bharath), at page 7, section "Value functions" (and subsection "Function Approximation and the DQN"), it's written

It was designed such that the final fully connected layer outputs $Q^\pi(s,\cdot)$ for all action values in a discrete set of actions — in this case, the various directions of the joystick and the fire button. This not only enables the best action, $\text{argmax}_a Q^\pi(s, a)$, to be chosen after a single forward pass of the network, but also allows the network to more easily encode action-independent knowledge in the lower, convolutional layers.