14

Actor-Critic is not just a single algorithm, it should be viewed as a "family" of related techniques. They're all techniques based on the policy gradient theorem, which train some form of critic that computes some form of value estimate to plug into the update rule as a lower-variance replacement for the returns at the end of an episode. They all perform "...


14

Just ignore the invalid moves. For exploration it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If you only punish illegal moves they will still retain some probability (however small) and therefore will be executed from time to time (however seldom). So you ...


13

Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and the features $\phi$ is a function of the current state $s$ and an action from the set of actions $A$. $$ \pi(\theta, a) = \frac{e^{\theta \phi(s, a)}}{\sum_{b \...


7

I faced a similar issue recently with Minesweeper. The way I solved it was by ignoring the illegal/invalid moves entirely. Use the Q-network to predict the Q-values for all of your actions (valid and invalid) Pre-process the Q-values by setting all of the invalid moves to a Q-value of zero/negative number (depends on your scenario) Use a policy of your ...


7

IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative reward. Absurd? Sure! But in fact your invalid moves are just a relic of the representation (which itself is straightforward and fine). The best treatment of them is to exclude them completely ...


7

There are many algorithms that are not based on finding a value function. The most common ones are policy gradients. These methods attempt to map states to actions through a neural network. They learn the optimal policy directly, not through a value function. The important part of the image is when Model-Free RL splits into Policy Optimization (which ...


5

One can expect the optimal high-level features required to choose the next action and to evaluate a state to be quite similar. Because of that, it is a reasonable idea to share the same network for both policy and value function – you are essentially parameter sharing the feature-extraction part of your neural network, and fine tuning the different heads of ...


5

In general, what are the advantages of RL with actor-critic methods over actor-only (or policy-based) methods? One practical benefit is that critics can use TD learning to bootstrap, allowing them to learn online on each step taken, plus learn in continuing problems. Pure actor algorithms like REINFORCE, cross-entropy method, and non-RL policy-only learners,...


4

For discrete action spaces, what is the purpose of the actor in Actor-Critic algorithms? In brief, it is the policy function $\pi(a|s)$. The critic (a state action function $v_{\pi}(s)$) is not used to derive a policy, and in "vanilla" Actor-Critic cannot be used in this way at all unless you have the full distribution model of the MDP. It just seems to ...


4

I think what you mean to ask is how can differentiation occur when there's no obvious neural network function to differentiate? Don't worry - lots of people get confused about this, because it seems like an obvious hole in the puzzle. As mentioned by @AtillaOzgur, neural networks use partial differentiation through backpropagation. First, take the output ...


4

The twist here is that the $a_{t+1}$ in (11) and the $\mu(s_{t+1})$ in (16) are the same and actually the $a_t$ in the on-policy case and the $a_t$ in the off-policy case are different. The key to the understanding is that in on-policy algorithms you have to use actions (and generally speaking trajectories) generated by the policy in the updating steps (to ...


4

When using the loss function for the critic described in your question, the Actor-Critic is an on-policy approach (as are most Actor-Critic methods). Your intuition as to what it is learning seems to be quite close, but the notation/terminology is not quite on point. First it's important to realize that the $Q(s, a)$ critic is an estimator, we're training ...


4

In the answer I'll be using notation similar to the one from the SAC paper. If we look at the standard objective function for policy gradient methods we have \begin{align} J_\pi &= V_\pi(s_t)\\ &= \mathbb E_{a_t \sim \pi(a|s_t)}[Q(s_t, a_t)]\\ &= \mathbb E_{a_t \sim \pi(a|s_t)}[ \mathbb E_{s_{t+1} \sim p(s|s_t, a_t)} [r(s_t, a_t) + V(s_{t+1})]]\\ ...


4

This is simply from definition of return in average reward setting (look at equation $10.9$). The "standard" TD error is defined as \begin{equation} TD_{\text{error}} = R_{t+1} + V(S_{t+1}) - V(S_t) \end{equation} In average reward setting, average reward $r(\pi)$ is subtracted from reward at $t$, $R_t$, so TD error in this case is \begin{equation} TD_{\text{...


4

Let's say your old policy is $\pi_b$ and your current one is $\pi_a$. If you collected trajectory by using policy $\pi_b$ you would get return $G$ whose expected value is \begin{align} E_{\pi_b}[G_t|S_t = s] &= E_{\pi_b}[R_{t+1} + G_{t+1}]\\ &= \sum_a \pi_b(a|s) \sum_{s', r} p(s', r|s, a) [r + E_{\pi_b}[G_{t+1}|S_{t+1} = s']]\\ \end{align} You can ...


4

MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance. Therefore, methods used in RL to solve MDPs will work to solve contextual bandits. You can either treat a contextual bandit as a series of 1-step episodes (with start state chosen randomly), or as a ...


3

I'll give it a go here and try to answer your question, I'm not sure if this is entirely correct, so if someone thinks that it isn't please correct me. I'll disregard expectation here to make things simpler. First, note that policy $\pi$ depends on parameter vector $\phi$ and function $f_\phi(\epsilon_t;s_t)$, and value function $Q$ depends on parameter ...


3

You're right, the first time you run it the two policies ($\pi_{\theta old}$ and $\pi_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\theta)={\pi_\theta(a|s)\over\pi_{\theta old}(a|s)}$) by the advantage (so $loss=-r_t(\theta)A_t$). However, with PPO you run multiple epochs of training on the ...


2

According to Sutton and Barto, they are the same thing. Note 13.5-6 (page 338) of their Reinforcement Learning: An Introduction, 2nd Edition book: Actor-critic methods are sometimes referred to as advantage actor-critic methods in the literature


2

An experimental paper exist in arxiv about the effect of whether to mask or to give negative rewards to invalid actions. There are some references in this paper which also discuss the effects and the mechanism to handle invalid actions. However, those main references are still only pre-prints in the arxiv (not published and presumably not peer-reviewed yet). ...


2

I believe if you run a single agent in multiple parallel environments many times you will get similar actions in similar states, the reason behind multiple agents is that you will have different agents with different parameters and you can also have different explicit exploration policies so your exploration will be better and you will learn more from ...


2

There are different actor-critic (AC) algorithms with different convergence guarantees. For example, AC algorithms where the critic is tabular have different convergence guarantees than AC algorithms where the critic is a neural network (function approximation). Most convergence proofs assume that the actor and the critic operate at different time scales, ...


2

Theoretically, nothing precludes the use of $\lambda$-returns in actor-critic methods. The $\lambda$-return is an unbiased estimator of the Monte Carlo (MC) return, which means they are essentially interchangeable. In fact, as discussed in High-Dimensional Continuous Control Using Generalized Advantage Estimation, using the $\lambda$-return instead of the MC ...


2

Keeping this taxonomy intact for model-based Dynamic programming algorithms, I would argue that value iteration is a Actor only approach, and policy iteration is a Actor-Critic approach. However, not many people discuss the term Actor-Critic when referring to Policy Iteration. How come? Both policy iteration and value iteration are value-based approaches. ...


2

Yeah, it seems like it's a wrong implementation. vals_ref_v is a matrix of 1 row, and 128 columns. value_v.detach() is a matrix of 128 row


2

Here is the commit I fixed few minor errors, but the major one was when I saw what the line histories = [deque(maxlen=self.reward_steps)] * len(self.env.envs) was doing. It was just repeating the same queue. In [2]: histories = [deque(maxlen=5)] * 4 In [3]: histories ...


2

Actually, your result that the gradient is 0 is correct given your formulation. Indeed, that is why one might have believed that the deterministic policy gradient didn't exist. The term $\nabla_{\theta}\log \pi(a \, | \, s)$ is a type of gradient estimator known as a likelihood ratio, and it assumes that the support of $\pi$ does not depend on $\theta$. In ...


1

Unfortunately no, the way to go is track the total reward and see if it's increasing and converging eventually. Value loss isn't a useful metric as the loss can be 0 when the value network always predicts 0 and the agent doesn't collect any reward, meaning very poor behavior.


1

The paper Dota 2 with Large Scale Deep Reinforcement Learning goes into greater detail than the initial blog posts. They call their distributed training framework Rapid, which is also used in some of their robotics work, such as the paper Learning Dexterous In-Hand Manipulation, where they discuss a smaller scale deployment of Rapid (as compared to Dota2/...


1

OpenAI have a post on that: https://openai.com/blog/openai-five/ They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed for batches of 4096 observations which are then averaged. PPO is actually designed to allow this kind of parallelisation as it uses trajectory segments with a ...


Only top voted, non community-wiki answers of a minimum length are eligible