# How do I calculate $max_{a′}Q(s′,a′,w−)$ when it is represented as a neural network?

Consider the following loss function

$$L(\mathbf{w}) = [(r + \gamma max_{a'} Q(s', a', \mathbf{w^-})) - Q(s, a, \mathbf{w})]^2$$

where $$Q(s, a, \mathbf{w^-})$$ and $$Q(s, a, \mathbf{w})$$ are represented as neural networks, where $$w^-$$ and $$w$$ are the corresponding weights.

But how do you calculate $$max_{a'} Q(s', a', \mathbf{w^-})$$? Do you really need to hold always an older version of the network? If yes, why and how old should it be?

You calculate the max by calculating your estimates for all possible actions for the next state, and taking the highest value.

The details depend a little on your neural network architecture:

• If you have a network that takes the state vector as input and outputs all possible action values $$(\hat{q}(s,a_0), \hat{q}(s,a_1), \hat{q}(s,a_2) ...)$$, then you can run it once forward with the next_state as input to getthe array $$(\hat{q}(s',a_0), \hat{q}(s',a_1), \hat{q}(s',a_2) ...)$$, and take the maximum element (you don't need to care which action $$a'$$ caused it). You will then have the problem that you now have a loss for $$Q(s, a, \mathbf{w})$$ for the single action $$a$$ just taken, but no data any of the alternative actions. The loss for these alternative actions needs to be set to zero - if you are training the NN using a normal supervised learning approach, that means you need to keep the full output of the network that you ran forward to calculate $$Q(s, a, \mathbf{w})$$ then substitute in this new estimated value against the action $$a$$ and train using this modified vector.

• If you have a network that takes the state vector and action combined as input and outputs a single estimate $$\hat{q}(s,a)$$, then you have to run that network once for each possible action from the next state and take the maximum value. You would typically do this as a small batch prediction for better performance. In this case your training data is simple to construct as you only have the loss for and train the network against one state/action combination.

Overall the first option (all action values at once) is usually a lot more efficient, but slightly more complex to code the training routine.

do you really need to hold always an older version of the network. If yes why and how old should it be?

You don't have to, but it is highly advisable to have this target network (so called because it helps generate your TD Target values), because Q learning using neural networks is often unstable. This is due to the bootstrap calculations where estimates are based on other estimates plus a little bit of observed data at each step. There is a strong possibility for runaway feedback due to training a neural network on something that includes its own output.

How old should it be? That's sadly a hyper-parameter of the architecture that you will need to establish through experiment on each new problem. I have worked with maximum age values from 100 to 10000 in my own simple experiments. Note this is not usually a rolling age - you don't keep 1000 copies of the network weights. Just keep one frozen copy, and after N steps replace it with a copy of the most recent one.

One alternative to this copy/freeze/copy approach is to update the target network towards the learning network on every step by a small factor. E.g. $$\mathbf{w}^{\_} = (1 - \beta)\mathbf{w}^{\_} + \beta \mathbf{w}$$ where $$\beta$$ might be $$0.001$$

In addition, you should be using experience replay for training data, and not training directly online. The combination of experience replay and using a frozen or slowly adapting target network makes a large difference to the stability of deep Q learning in practice.

• I'd describe the need for target networks being due to the "moving target" problem that you otherwise get, rather than numeric stability. Numeric stability can easily be confused with problems/inaccuracies related to floating point arithmetic, which is not really the problem addressed by target networks (although I suppose in some cases the moving target instability may get so severe that reaching the realms of NaN becomes a symptom... but that's not the only possible symptom I think?) Commented Jan 5, 2019 at 19:17
• So what wrong with the Brain.class (I did exactly this except experience replay) github.com/SuchtyTV/RLearningBird/blob/master/src/main/java/… Commented Jan 5, 2019 at 19:44
• @TVSuchty: Sorry I am not going to try and debug your project. I don't work in Java so it would take me too long and I may miss any obvious problem. Commented Jan 5, 2019 at 20:25
• Could you help me if i translate it into pseudocode? or python? Commented Jan 5, 2019 at 20:29
• @TVSuchty: Sorry no. If you can write a question about some part of the code that you are having trouble with, then I or someone else here might be able to help you with an answer. One thing you could do is write out a brief high-level pseudo-code for what your Brain class is doing, and ask if that is correct in a question, or if you have implemented it and it is not working, ask about the problems you are having. There are already many examples of working DQN in Python available though if you search . . . Commented Jan 5, 2019 at 20:31