0
$\begingroup$

I'm trying to implement the DQN paper using python/pytorch for my needs (https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf). I'm studying the main algorithm: enter image description here

I am a bit confused about the $\gamma* \max Q$ when setting the $y$. My model essentially takes a state and outputs a single float value (reward, cost, call it whatever). Do I have to calculate the Q for every possible action in my action space for that particular state or simply the Q of the state?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

Do I have to calculate the Q for every possible action in my action space

Yes, for the given state vector $\phi_{j+1}$ you must calculate the action value for all possible actions from that state. That is the only way that you can calculate $\text{max}_{a'} Q(\phi_{j+1}, a'; \theta)$ used to set the TD target $y_j$.

or simply the Q of the state?

That is not defined. Q is an action value, and can only be calculated when you know both the state and the action.

In practice, your neural network in DQN will be one of two designs:

  • It takes state and action concatenated as input and models the state-action value function directly: $Q(s,a): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$. In this case you will need to run a mini-batch of all allowed actions in the same state in order to find $\text{max}_{a'} Q(\phi_{j+1}, a'; \theta)$

  • It takes state as input and models the state-action value for all actions at once: $f(s): \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$. In this case you will only need to run the forward action once to find $\text{max}_{a'} Q(\phi_{j+1}, a'; \theta)$, but you will need to manipulate the resulting TD target data to ensure the only error gradient is from the selected action - typically this is done by running the network forward on the current state $\phi_{j}$ and modifying that vector with the new TD taget only for the taken action $a_j$

These designs are not mentioned in the pseudo-code because they are an implementation detail somewhat separate from the theory. You will find the second design is more common in DQN implementations, as it often has better performance (in terms of raw speed processing each state).

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you. Truth be told, my implementation is V-value rather than Q-value for various reasons, thus the confusion on my part. $\endgroup$
    – Antonis Karvelas
    Apr 14, 2022 at 12:30
  • $\begingroup$ @AntonisKarvelas You can use state value V if you are using a model and/or afterstates (which are a kind of hybrid state/action and common in e.g. board and card games)- in which case you will need to code to look ahead to get all possible $V(s')$ depending on the action or afterstate choices. You will almost definitely be using the first architecture in your case. $\endgroup$
    – Neil Slater
    Apr 14, 2022 at 12:37
  • $\begingroup$ I'm doing exactly this. Is there a paper or an implementation I can look into? So far I'm following my idea of this hybrid and I may be doing something silly. $\endgroup$
    – Antonis Karvelas
    Apr 14, 2022 at 12:39
  • $\begingroup$ @AntonisKarvelas I am sure there will be some implentations using DQN online, as well as various MCTS/"Alpha Zero" variations. For just the Q-learning side I produced a tic tac toe learner in Python and published in on Kaggle here: kaggle.com/code/slobo777/tic-tac-toe-agent-using-q-learning (the original repo I deleted, as I never filled in with planned other games) $\endgroup$
    – Neil Slater
    Apr 14, 2022 at 12:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .