8

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. In many cases this one distribution is enough because only 1 continuous action is required. However, as domains such as robotics become more integrated with AI, ...


6

I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story: In the PPO paper, after describing the standard policy gradient objective $L^{PG}$, they mention the following: While it is appealing to perform multiple steps of optimization on this loss $L^{PG}$ using the same trajectory, doing so ...


6

The most straightforward solution is to simply make every action "legal", but implementing a consistent, deterministic mapping from potentially illegal actions to different legal actions. Whenever the PPO implementation you are using selects an illegal action, you simply replace it with the legal action that it maps to. Your PPO algorithm can then still ...


4

Both implementations may be closer than you think. In short: PPO has both parts: there is noisiness in draws during training (with learned standard deviation), helping to explore new promising actions/policies. And there is a term added to the loss function aiming to prevent a collapse of the noisiness, to help ensure exploration continues and we don't get ...


4

Do parallel environments improve the agent's ability to learn or does it not really make a difference? Yes they can make a difference. There are two ways improvement is seen: Collecting data from multiple trajectories at once reduces correlation in the dataset. This improves convergence for online learning systems like neural networks, which work best with ...


4

We do decorrelate training experience, even for policy gradient methods. This is because decorrelation helps training data be more like IID data, which helps with the convergence of SGD-like optimizers. The shuffling is done on line 151 of OpenAI's "baselines" implementation of PPO. I'm going to guess that there's a bug somewhere in your implementation. ...


3

No, we typically don't use a validation/test data set in Reinforcement Learning (RL). This is because of how we use the data in RL. The use of a data set is very different to the classic supervised/unsupervised paradigms. Some RL algorithms don't even have a data-set as such. For instance, the vanilla tabular Q-learning does not use a data-set -- it will see ...


3

Personally, I would choose the following two as the most important: epsilon: When using an epsilon-greedy policy, epsilon determines how often the agent should explore and how often it should exploit. Balancing exploration and exploitation is crucial for the success of the learning agent. Too little exploration might not teach anything to the agent and too ...


3

For everybody getting here from google, like me: the $\log$ might have been replaced in the loss function, but I think it is still there when taking the gradient of both functions (correct me, if I am wrong): $$\begin{aligned} \nabla_{\theta} L^{P G}(\theta) &=\nabla_{\theta} \hat{E}_{t}\left[\log \pi_{\theta}\left(a_{t} \mid s_{t}\right) \hat{A}_{t}\...


3

You're right, the first time you run it the two policies ($\pi_{\theta old}$ and $\pi_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\theta)={\pi_\theta(a|s)\over\pi_{\theta old}(a|s)}$) by the advantage (so $loss=-r_t(\theta)A_t$). However, with PPO you run multiple epochs of training on the ...


2

Change the action space at each step, depending on the internal_state. I assume this is nonsense. Yes, this seems overkill and makes the problem unnecessarily complex, there could be other things you can do. Do nothing : let the model understand that choosing an unavailable action has no impact. While this will not harm your model negatively, in any way ...


2

Loss. In the context of Deep Learning and Deep Reinforcement Learning, "training" is just a fancy word for "optimization". You are essentially looking for an optimum point in some huge-dimensional parameter space. For the Reinforcement Learning problems, the optimal point is supposed to maximize the expected return - while most of the ...


2

The answer to both your concerns is: Add the previous action choice to the state representation. It is all you need to do. It gives the agent the data it needs to learn the association of negative reward from not matching the previous action. By making this data part of the state, you re-establish the Markov property in the MDP model of the environment, ...


2

As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :) A common example for continuous spaces is the reparameterization trick, where your policy outputs $\mu, \sigma = \pi(s)$ and the action is $a \sim \mathcal{N}(\mu, \sigma)$.


2

The idea in PPO is that you want to reuse the batch many times to update the current policy. However, you cannot update mindlessly in a regular actor-critic fashion, because your policy might stray too far away from the optimal point. This means you repeat your step 6. epoch amount of times for the same batch of trajectories. Usually epoch is somewhere ...


2

You should read this study https://arxiv.org/abs/2006.05990 which does some empirical study on this question, specifically for on-policy, continuous action space DRL. It suggests that discount factor and learning rate are the two most important parameters to tune, followed by the width of the policy/value functions. That study also reports that it's very ...


1

A different, variable reward structure might help. You could try a combination of airspeed, pitch, roll and whether it is hovering in the air or not in each timestep as a representation for the reward. Maybe airspeed should, in expectation, contribute up to 30% of the reward, pitch up to 15%, roll up to 15% and being in the air up to 40%. This would ...


1

Training using only 20 timesteps at a time is far too small, especially when the goal will ultimately consist of episodes of length 6000. You definitely need to increase that substantially and that will probably solve your problem immediately. You might try something like simulate 5 episodes and then train on all timesteps in those 5 episodes. If that still ...


1

First of all, the support of a normal distribution is the entire real line (or, in general, $\mathbb{R}^n$ for an $n$-dimensional multivariate normal distribution) so your action can be any number in $\mathbb{R}$. What you may be getting confused with is that with probability 0.68 you will obtain an action that is within +/- 1 standard deviation from the ...


1

As you point out, they are not equivalent. I guess you could store the time index for each state visited, but there are two problems with this. First, if you sample states according to their time index, sampling from the replay memory will become more cumbersome and probably much slower (you'd have to sample the time index and then a specific state with ...


1

I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.


1

I had the same problem where the reward kept decreasing and started to search for answers in the forum. I let the model trained while I search. As the model trained, the reward started to increase. You can see the tensorboard graph for rewards in validation time. . The fall continued until around 100k~ steps and did not change a lot for 250k~ steps. ...


1

The paper Dota 2 with Large Scale Deep Reinforcement Learning goes into greater detail than the initial blog posts. They call their distributed training framework Rapid, which is also used in some of their robotics work, such as the paper Learning Dexterous In-Hand Manipulation, where they discuss a smaller scale deployment of Rapid (as compared to Dota2/...


1

OpenAI have a post on that: https://openai.com/blog/openai-five/ They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed for batches of 4096 observations which are then averaged. PPO is actually designed to allow this kind of parallelisation as it uses trajectory segments with a ...


1

I found the error: division by zero when calculating the ratio, here. ratio = pi.prob(self.tfa) / oldpi.prob(self.tfa) I changed to: ratio = tf.divide(pi.prob(self.tfa), tf.maximum(oldpi.prob(self.tfa), 1e-5))


1

Normally, the set of actions that the agent can execute does not change over time, but some actions can become impossible in different states (for example, not every move is possible in any position of the TicTacToe game). Take a look as example at pice of code https://github.com/haje01/gym-tictactoe/blob/master/examples/base_agent.py : ava_actions = env....


Only top voted, non community-wiki answers of a minimum length are eligible