# Continuous Advantage Actor Critic Implementation

I'm having trouble implementing AC for continuous action space. As far as I can tell, my code doesn't seem to have any bugs! The agent is learning "something" as its behaviour seems to vary dramatically after several episodes, but it never seems to ever approach a type of behaviour which I'd think is reasonable.

I've used very similar code and things have gone smoothly in discrete space and little has changed other than changes to the output (mean and variance).

Below is the relevant code:

class Actor(object):
def __init__(self, sess, s_size, h_size, a_size, env, lr=1e-3):

mu = tf.layers.dense(self.hidden_1,
self.a_size,
activation=tf.nn.tanh,
bias_initializer=None)

sigma = tf.layers.dense(self.hidden_1,
self.a_size,
activation=tf.nn.softplus,
bias_initializer=None)
sigma = sigma + 1e-10

self.normal_dist = tf.contrib.distributions.Normal(mu, sigma)
self.action = tf.clip_by_value(self.normal_dist.sample(1), env.action_space.low[0], env.action_space.high[0])

self.adv = tf.placeholder(dtype=tf.float32)    # get log prob of the actions taken in _samples
self.acts = tf.placeholder(shape=[None, a_size], dtype=tf.float32)

self.log_prob = self.normal_dist.log_prob(self.acts)

self.loss = -self.log_prob * self.adv


The environment I'm using is the LunarLanderContinuous-v2. I've tested DDPG in this same environment and the agent learns incredibly quickly in comparison with the same learning rate and model size which is making me very confused. If anyone has any input it would be very much appreciated. Thanks

class Critic(object):
def __init__(self, sess, s_size, h_size, env, gamma=0.99, lr=1e-3):
self.gamma = gamma
self.replay_buffer = []
self.input = tf.placeholder(shape=[None, s_size], dtype=tf.float32)

self.hidden_1 = tf.layers.dense(self.input,
h_size,
activation=tf.nn.relu,
bias_initializer=None)

self.hidden_2 = tf.layers.dense(self.hidden_1,
h_size,
activation=tf.nn.relu,
bias_initializer=None)

self.value = tf.layers.dense(self.hidden_2,
1,
activation=None,
bias_initializer=None)

self.q_value = tf.placeholder(shape=[None,], dtype=tf.float32)
self.advantage = self.q_value - self.value

self.loss = tf.reduce_mean(tf.square(self.advantage))

self.lr = lr
optimizer = tf.train.AdamOptimizer(self.lr)
self.update = optimizer.minimize(self.loss)

• As you have only posted the Actor's constructor there is no way to tell where the problem may lie. Furthermore, note that vanilla AC methods are not very sample-efficient... – geky Oct 30 '18 at 17:54
• @geky I showed the Actor part because I've tested the critic part in other environments and am using the same model. I've posted it now. Do you have any recommendation for other AC methods that are not vanilla or deterministic like DDPG? – tryingtolearn Oct 31 '18 at 10:39
• Is the code in your question the code you're actually running? Inside the Actor class, you're using self.hidden_1... but that doesn't seem to be defined there? A variable with that name is only defined in the different Critic class – Dennis Soemers Nov 4 '18 at 11:13
• @DennisSoemers It's not, I edited out most of the stuff before because the variables were exactly the same as previously so I confirm they're working. I just posted the stuff that I had to adapt which is outputing the mu and sigma for the distributions which has 2 dimensions. – tryingtolearn Nov 5 '18 at 0:44
• How are you calculating and feeding in advantage? I can see you are setting a placeholder to feed it in later self.q_value = tf.placeholder(shape=[None,], dtype=tf.float32) but not how you are calculating it. – Neil Slater Nov 6 '18 at 14:04

## 1 Answer

At first, let us define some vocabulary. The LunarLanderContinuous-v2 example is an optimal control problem. A nonlinear system gets input from the user and as a result the lander is doing actions. The actor critic implementation is a special type in reinforcement learning which works by a state action table. The inner core of the algorithm is a qlearning table. It is a matrix structure aka “lookup table” in which for each possible situation of the game the appropriate reaction is stored. What the Actor critic algorithm is doing is to change the values inside the q-table in respect to the reward. The state-space of all possible q-tables has to be searched.

If the actor-critic method fails, that is equal to not finding the correct q-table. There are many possible reasons for that. One option is, that the state space is too big, that means the number of states is too high. That seems logic, because it is a continuous control problem which has by law a large state space. A second reason to fail is, that a q-table in general doesn't fit to the optimal control problem. That means, that reinforcement learning is the wrong idea. But hopefully the second case is wrong, because in the internet some example are available in which the lunarlander is serving great, so i would guess that the state space has only to reduced a bit.

A possible bugtracking option would be to print out the q-table to the screen during the learning process, so that the modification of the matrix becomes visible. This would give better feedback than only print out the error rate. This can be done in Tensorflow with “tf.print” or “tf.dump” but I'm not sure.

• My problem isn't to do with understanding the algorithm. It's my implementation. I've tested this problem many times and applied the same algorithm using discrete space and the LunarLander is extensively tested so this should work. For this reason I feel my issue is in my implementation of the sampling method for actions so that's where I'm looking for help. – tryingtolearn Nov 2 '18 at 12:23
• @tryingtolearn Sorry, it's my fault. I've tried out to downvote my own answer, but it didn't work. If you could ... – Manuel Rodriguez Nov 2 '18 at 12:38
• Not a problem at all! Thank you for trying to answer nonetheless. – tryingtolearn Nov 2 '18 at 15:14