3
$\begingroup$

In the classical examples of deep q-learning, I often see neural networks in which the input represents the state of the agent, while the output is a tuple with all the values of $Q(s, a)$ predicted for all the possible $N$ actions.

Would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?

With cheaper I mean cheaper in terms of the time complexity of a single training step of the network.

$\endgroup$
8
  • $\begingroup$ Hello. I tried to clarify your question. Make sure that I didn't change the meaning of the post. Please, could you also clarify what you mean by "cheaper"? Cheaper in terms of what? $\endgroup$
    – nbro
    Apr 28 at 10:04
  • $\begingroup$ Hi, thank you. The meaning is the same. With "cheaper" I mean cheaper in terms of operations required. $\endgroup$ Apr 28 at 13:21
  • $\begingroup$ Please, edit your post to clarify what you mean by "cheap". In terms of operations, do you mean the time complexity of one iteration of training DQN as described in the original paper? $\endgroup$
    – nbro
    Apr 29 at 10:27
  • $\begingroup$ Yes, I mean that. I have edited. $\endgroup$ Apr 29 at 20:18
  • $\begingroup$ But, if that's what you mean, why did you accept the answer below? It doesn't really answer your question (which was not specific enough indeed). You should discuss your edited version with the author of that answer. $\endgroup$
    – nbro
    Apr 29 at 22:27
3
$\begingroup$

Would it be cheaper to have $N$ neural networks with a single real-valued output, one for each of the $N$ actions?

I think the "No Free Lunch" theorem applies here, or something like it.

Your proposed architecture would be an unusual choice in many cases, but might be more efficient in others. For instance, it could be more efficient in the following scenario:

The long term value is highly dependent on the immediate action choice, and in a way that relies on state variables differently, depending on the specific action. That means it would be difficult for a single NN to create shared features in its layers, and you could save processing by treating each action as a different prediction problem.

This is only an educated guess.

As usual, the only way to find out for sure is to try different approaches and compare them. I don't think there is anything other than experience and a little intuition to guide you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.