7
As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. In many cases this one distribution is enough because only 1 continuous action is required. However, as domains such as robotics become more integrated with AI, ...
6
I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story:
In the PPO paper, after describing the standard policy gradient objective $L^{PG}$, they mention the following:
While it is appealing to perform multiple steps of optimization on this loss $L^{PG}$ using the same trajectory, doing so ...
6
The most straightforward solution is to simply make every action "legal", but implementing a consistent, deterministic mapping from potentially illegal actions to different legal actions. Whenever the PPO implementation you are using selects an illegal action, you simply replace it with the legal action that it maps to. Your PPO algorithm can then still ...
4
Do parallel environments improve the agent's ability to learn or does it not really make a difference?
Yes they can make a difference. There are two ways improvement is seen:
Collecting data from multiple trajectories at once reduces correlation in the dataset. This improves convergence for online learning systems like neural networks, which work best with ...
3
We do decorrelate training experience, even for policy gradient methods. This is because decorrelation helps training data be more like IID data, which helps with the convergence of SGD-like optimizers.
The shuffling is done on line 151 of OpenAI's "baselines" implementation of PPO.
I'm going to guess that there's a bug somewhere in your implementation. ...
3
You're right, the first time you run it the two policies ($\pi_{\theta old}$ and $\pi_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\theta)={\pi_\theta(a|s)\over\pi_{\theta old}(a|s)}$) by the advantage (so $loss=-r_t(\theta)A_t$).
However, with PPO you run multiple epochs of training on the ...
3
Personally, I would choose the following two as the most important:
epsilon: When using an epsilon-greedy policy, epsilon determines how often the agent should explore and how often it should exploit. Balancing exploration and exploitation is crucial for the success of the learning agent. Too little exploration might not teach anything to the agent and too ...
2
You should read this study https://arxiv.org/abs/2006.05990 which does some empirical study on this question, specifically for on-policy, continuous action space DRL.
It suggests that discount factor and learning rate are the two most important parameters to tune, followed by the width of the policy/value functions.
That study also reports that it's very ...
2
Change the action space at each step, depending on the internal_state. I assume this is nonsense.
Yes, this seems overkill and makes the problem unnecessarily complex, there could be other things you can do.
Do nothing : let the model understand that choosing an unavailable action has no impact.
While this will not harm your model negatively, in any way ...
2
Both implementations may be closer than you think.
In short:
PPO has both parts: there is noisiness in draws during training (with learned standard deviation), helping to explore new promising actions/policies. And there is a term added to the loss function aiming to prevent a collapse of the noisiness, to help ensure exploration continues and we don't get ...
2
Loss. In the context of Deep Learning and Deep Reinforcement Learning, "training" is just a fancy word for "optimization". You are essentially looking for an optimum point in some huge-dimensional parameter space. For the Reinforcement Learning problems, the optimal point is supposed to maximize the expected return - while most of the ...
2
The answer to both your concerns is:
Add the previous action choice to the state representation.
It is all you need to do. It gives the agent the data it needs to learn the association of negative reward from not matching the previous action.
By making this data part of the state, you re-establish the Markov property in the MDP model of the environment, ...
2
As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :)
A common example for continuous spaces is the reparameterization trick, where your policy outputs $\mu, \sigma = \pi(s)$ and the action is $a \sim \mathcal{N}(\mu, \sigma)$.
2
The idea in PPO is that you want to reuse the batch many times to update the current policy. However, you cannot update mindlessly in a regular actor-critic fashion, because your policy might stray too far away from the optimal point.
This means you repeat your step 6. epoch amount of times for the same batch of trajectories. Usually epoch is somewhere ...
1
As you point out, they are not equivalent. I guess you could store the time index for each state visited, but there are two problems with this.
First, if you sample states according to their time index, sampling from the replay memory will become more cumbersome and probably much slower (you'd have to sample the time index and then a specific state with ...
1
I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.
1
I had the same problem where the reward kept decreasing and started to search for answers in the forum.
I let the model trained while I search. As the model trained, the reward started to increase. You can see the tensorboard graph for rewards in validation time.
.
The fall continued until around 100k~ steps and did not change a lot for 250k~ steps.
...
1
The paper Dota 2 with Large Scale Deep Reinforcement Learning goes into greater detail than the initial blog posts.
They call their distributed training framework Rapid, which is also used in some of their robotics work, such as the paper Learning Dexterous In-Hand Manipulation, where they discuss a smaller scale deployment of Rapid (as compared to Dota2/...
1
OpenAI have a post on that: https://openai.com/blog/openai-five/
They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed for batches of 4096 observations which are then averaged.
PPO is actually designed to allow this kind of parallelisation as it uses trajectory segments with a ...
1
I found the error: division by zero when calculating the ratio, here.
ratio = pi.prob(self.tfa) / oldpi.prob(self.tfa)
I changed to:
ratio = tf.divide(pi.prob(self.tfa), tf.maximum(oldpi.prob(self.tfa), 1e-5))
1
Normally, the set of actions that the agent can execute does not change over time, but some actions can become impossible in different states (for example, not every move is possible in any position of the TicTacToe game).
Take a look as example at pice of code https://github.com/haje01/gym-tictactoe/blob/master/examples/base_agent.py :
ava_actions = env....
1
For everybody getting here from google, like me: the $\log$ might have been replaced in the loss function, but I think it is still there when taking the gradient of both functions (correct me, if I am wrong):
$$\begin{aligned}
\nabla_{\theta} L^{P G}(\theta) &=\nabla_{\theta} \hat{E}_{t}\left[\log \pi_{\theta}\left(a_{t} \mid s_{t}\right) \hat{A}_{t}\...
Only top voted, non community-wiki answers of a minimum length are eligible
