7

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. In many cases this one distribution is enough because only 1 continuous action is required. However, as domains such as robotics become more integrated with AI, ...


6

I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story: In the PPO paper, after describing the standard policy gradient objective $L^{PG}$, they mention the following: While it is appealing to perform multiple steps of optimization on this loss $L^{PG}$ using the same trajectory, doing so ...


5

We can start with equation (30): $$ \bar{A}(s) = P(a \neq \tilde{a}) \mathbb{E}_{(a,\tilde{a})\sim(\pi,\tilde{\pi}|a\neq\tilde{a})} [A_\pi(s, \tilde{a}) - A_\pi(s, a)] $$ Taking the absolute value of both sides, the equality remains true. We can pull the probability term out of the absolute value since it is guaranteed to be nonnegative. $$ |\bar{A}(s)| = ...


3

The differences you have observed between the two different versions of the TRPO paper are due to different formalizations of the problem and the objective. In the first version of the paper you linked, they start out in Section 2 by defining Markov Decision Processes (MDPs) as tuples that, among other things, have a cost function $c : \mathcal{S} \...


1

As you point out, they are not equivalent. I guess you could store the time index for each state visited, but there are two problems with this. First, if you sample states according to their time index, sampling from the replay memory will become more cumbersome and probably much slower (you'd have to sample the time index and then a specific state with ...


1

For everybody getting here from google, like me: the $\log$ might have been replaced in the loss function, but I think it is still there when taking the gradient of both functions (correct me, if I am wrong): $$\begin{aligned} \nabla_{\theta} L^{P G}(\theta) &=\nabla_{\theta} \hat{E}_{t}\left[\log \pi_{\theta}\left(a_{t} \mid s_{t}\right) \hat{A}_{t}\...


Only top voted, non community-wiki answers of a minimum length are eligible