New answers tagged

0

Both your notation and terminology are quite confusing. For example, I'm not sure what is an "optimal" Bellman operator is. Here's a good clarification on definition of a Bellman operator. Likewise, your description of the DQN algorithm completely ignores the averaging over states/actions/rewards sampled from the replay memory. Trying to savage ...


2

Let's add a step index to your expression $$Q_{target}^{n} = (1-\tau)Q^{n-1}_{target} + \tau\, Q^{n-1}_{primary}$$ We can expand it one step further $$Q_{target}^{n} = (1-\tau)^2Q^{n-2}_{target} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$ And further $$Q_{target}^{n} = (1-\tau)^3Q^{n-3}_{target} + (1-\tau)^2\tau\, Q^{n-3}_{primary} + (1-\...


0

I found the reason it wasn't learning. The issue was this line of code: q_target[torch.arange(states.size()[0]), actions] = rewards + (self.gamma * next_q_vals.max(dim=1)[0]) * (~dones).float() I had been using the tilde operator before to invert uint8 tensors, but recently I had updated to the latest version of pytorch that seems to have changed how the ...


1

Regarding your first question, $$V^{\pi}(s) = \sum_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$ so recovering the value function from Q really depends on what policy $\pi$ you are using. Hence, you can't really recover the value function $V(s)$ from the $Q(s,a)$ values without knowing your policy distribution for state $s$. However, you can recover $Q^{\pi}(s,a)$ values ...


1

In my experience, neural networks with convolutional layers take much much longer to train, so try increasing the number of iterations (time steps). After running, save the network model (I dont know how to do it in torch, but in tensorflow it was model.save("filename"+".h5") ). Then, load this saved model file and do a test run to see if ...


Top 50 recent answers are included