In the classical examples of deep q-learning, I often see neural networks in which the input represents the state of the agent, while the output is a tuple with all the values of $Q(s, a)$ predicted for all the possible $N$ actions.
Would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?
With cheaper I mean cheaper in terms of the time complexity of a single training step of the network.
Would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?
I think the "No Free Lunch" theorem applies here, or something like it.
Your proposed architecture would be an unusual choice in many cases, but might be more efficient in others. For instance, it could be more efficient in the following scenario:
The long term value is highly dependent on the immediate action choice, and in a way that relies on state variables differently, depending on the specific action. That means it would be difficult for a single NN to create shared features in its layers, and you could save processing by treating each action as a different prediction problem.
This is only an educated guess.
As usual, the only way to find out for sure is to try different approaches and compare them. I don't think there is anything other than experience and a little intuition to guide you.
