New answers tagged deep-rl
3
The AlphaZero paper mentions an "evaluation" step that seems to deal with the the problem similar to yours:
... we evaluate each new neural network checkpoint against the current best network $f_{\theta_*}$ before using it for data generation ... Each evaluation consists of 400 games ... If the new player wins by a margin of > 55% (to avoid ...
2
Let's add a step index to your expression
$$Q_{target}^{n} = (1-\tau)Q^{n-1}_{target} + \tau\, Q^{n-1}_{primary}$$
We can expand it one step further
$$Q_{target}^{n} = (1-\tau)^2Q^{n-2}_{target} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$
And further
$$Q_{target}^{n} = (1-\tau)^3Q^{n-3}_{target} + (1-\tau)^2\tau\, Q^{n-3}_{primary} + (1-\...
1
There is no sign error and we should not change to $\arg\max$. With Policy Gradients I find that it is not useful to think about things such as a 'loss'.
In short, we want to first find the derivative of the RL objective $J(\theta) = v_\pi(s_0)$, where $\pi$ is our policy that depends on some parameters $\theta$. The policy gradient theorem tells us that
$$\...
4
Yes, there are algorithms that try to predict the next state. Usually this will be a model based algorithm -- this is where the agent tries to make use of a model of the environment to help it learn. I'm not sure on the best resource to learn about this but my go-to recommendation is always the Sutton and Barto book.
This paper introduces PlanGAN; the idea ...
2
Check out Imagination-Augmented Agents paper - seems like it does what you are talking about. The agent itself is the standard A3C that you are familiar with. The novelty is the "imagination" environment model which is trained to predict the behavior of the environment.
1
When the authors write go from
$$\nabla_{\theta}J \propto \sum_s \mu(s) \sum_a q_{\pi}(s,a)\nabla_{\theta}\pi(a|s;\theta)\;$$
to
$$\nabla_{\theta}J = E_{\pi}\left[\sum_a q_{\pi}(S_t,a) \nabla_{\theta}\pi(a|S_t;\theta)\right]\;$$
they are simply taking an expectation where the only random variable is the state $S_t$. This is because, as they say in the book, ...
1
RL is currently being applied to environments which are definitely not markovian, maybe they are weakly markovian with decreasing dependency.
You need to provide details of your problem, if it is 1 step then any optimization system can be used.
0
Tl;dr max-pool
You can see in the diagram, everywhere there are a variable number of inputs (pickups, units, hero modifiers/abilities/items), a max-pool follows, though I don't know the specifics of the max-pool implementation.
From https://neuro.cs.ut.ee/the-use-of-embeddings-in-openai-five :
Notice that while the number of modifiers, abilities and items ...
0
You are right, it is sloppy notation by the authors. However, the target network is not necessarily linked to the behaviour policy $\beta$ either.
Essentially when they take the expectation with respect to $\rho^\beta$ they are taking expectation with respect to a state distribution induced by some policy $\beta$ that is not necessarily the same as our ...
Top 50 recent answers are included
