I recently started looking into implementations of the DQN algorithm (e.g. TensorFlow) in some more detail. All the implementations that I found use a network that gives an output for each possible action (e.g. if you have three possible actions you will have three output units in your network). This makes a lot of sense from a computational standpoint and seems to work fine if you are dealing with categorical action spaces (e.g "left" or "right").

However, I am currently working with an action space that I discretized and the actions have an ordinal meaning (e.g. you can drive left or right in 5-degree increments). I assume that the action-value function has some monotonicity in the action component (think driving 45 degrees to the left instead of 40 will have a similar value).

Am I losing information on the similarity of actions, if I use a network that has an output unit for each possible action?

Are there implementations of the DQN available in which actions are used as network inputs?

  • $\begingroup$ I tried to summarise what your main problem is by putting a question that attempts to include all your other questions in the title. Please, make sure that your main question is the one in the title. If it's not, please, feel free to edit your post again to put in the title your main question. $\endgroup$
    – nbro
    Feb 5 at 14:32

Yes it is possible to use the action as input to neural network in DQN. For discrete actions represented as one-hot encoded features, the difference is minor:

  • If all actions are in the output, your neural network function is $f(s): \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|} = [\hat{q}(s,a_1), \hat{q}(s,a_2), \hat{q}(s,a_3) ...]$, and you take the maximum value from the output vector as the greedy action.

  • If the action is provided as an input argument, your neural network function is $f(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \hat{q}(s,a)$, and to find the maximum value you construct and run a mini-batch over all possible values of $a$ for the given state.

Also see the answer to this question: Why does Deep Q Network outputs multiple Q values?

In your case, you would like to take advantage of similar values of $a$ because you expect that to work well with the approximation. As you correctly suggested, this will only work with the second approach using action as an input. So, use the steering angle, normalised into a suitable range for input to a neural network, as an input. Every time that you need to find $\text{max}_a Q(s,a)$ for the Q-learning algorithm, you must construct a mini-batch of the current state concatenated with each of the discrete steering angles that you want to consider as actions in the DQN, and run the current (or target) neural network forward.

If you want to go further and use a continuous action space, you will need to change which reinforcement learning method you are using. The various policy gradient and actor-critic approaches, such as REINFORCE, A3C, DDPG etc can cope with continuous actions, because they drop the need to find $\text{max}_a Q(s,a)$, which becomes impractical for very large action spaces.

  • $\begingroup$ Thank you Neil! I would like to upvote your answer but unfortunately miss the reputation. Are you by any chance aware of any library that provides an implementation of the second approach? I assume if I try to implement it, it will probably lead to some inefficiencies. $\endgroup$
    – PontrYagin
    Feb 5 at 9:34
  • $\begingroup$ @PontrYagin No sorry I do not know of any library for this. So far I have implemented all my DQNs directly in Keras or PyTorch. $\endgroup$ Feb 5 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.