# How many episodes does it take for a vanilla one-step actor-critic agent to master the OpenAI BipedalWalker-v2 problem?

I'm trying to solve the OpenAI BipedalWalker-v2 by using a one-step actor-critic agent. I'm implementing the solution using python and tensorflow.

I'm following this pseudo-code taken from the book Reinforcement Learning An Introduction by Richard S. Sutton and Andrew G. Barto. in summary, my question can be reduced to the following:

• Is it a good idea to implement a one-step actor-critic algorithm to solve the OpenAI BipedalWalker-v2 problem? If not what would be a good approach? If yes; how long would it take to converge?
• I run the algorithm for 20000 episodes, each episode has an avg of 400 steps, for each step, I immediately update the weights. The results are not better than random. I have tried different standard deviations (for my normal distribution that represents pi), different NN sizes for the Critic and Actor, and different learning-steps for the optimizer algorithm. The results never improve. I don't know what I'm doing wrong.

# My Agent Class

import tensorflow as tf
import numpy as np
import gym
import matplotlib.pyplot as plt

class agent_episodic_continuous_action():
def __init__(self, lr,gamma,sample_variance, s_size,a_size,dist_type):
... #agent parameters

def save_model(self,path,sess):
def weights_init_actor(self,hidd_layer,mean,stddev): #to have control over the weights initialization
def weights_init_critic(self,hidd_layer,mean,stddev):  #to have control over the weights initialization
def create_actor_brain(self,hidd_layer,hidd_act_fn,output_act_fn,mean,stddev):  #actor is represented by a fully connected NN
def create_critic_brain(self,hidd_layer,hidd_act_fn,output_act_fn,mean,stddev): #critic is represented by a fully connected NN
def critic(self):
def get_delta(self,sess):
def normal_dist_prob(self): #Actor pi distribution is a normal distribution whose mean comes from the NN
def create_actor_loss(self):
def create_critic_loss(self):
def sample_action(self,sess,state): #Sample actions from the normal dist. Whose mean was aprox. By the NN
def update_actor_weights(self):
def update_critic_weights(self):
def update_I(self):
def reset_I(self):
def update_time_step_info(self,s,a,r,s1,d):
def create_graph_connections(self):
def bound_actions(self,sess,state,lower_limit,uper_limit):


# Agent instantiation

tf.reset_default_graph()
agent= agent_episodic_continuous_action(learning-step=1e-3,gamma=0.99,pi_stddev=0.02,s_size=24,a_size=4,dist_type="normal")
agent.create_actor_brain(hidden_layers=[12,5],hidden_layers_fct="relu",output_layer="linear",mean=0.0,stddev=0.14)
agent.create_critic_brain(hidden_layers=[12,5],hidden_layers_fct="relu",output_layer="linear",mean=0.0,stddev=0.14)
agent.create_graph_connections()

path = "/home/diego/Desktop/Study/RL/projects/models/biped/model.ckt"
env = gym.make('BipedalWalker-v2')
uper_action_limit = env.action_space.high
lower_action_limit = env.action_space.low
total_returns=[]


# Training loops

with tf.Session() as sess:
try:
sess.run(agent.init)
sess.graph.finalize()
for i in range(1000):
agent.reset_I()
s = env.reset()
d = False
while (not d):
a=agent.bound_actions(sess,s,lower_action_limit,uper_action_limit)
s1,r,d,_ = env.step(a)
#env.render()
agent.update_time_step_info([s],[a],[r],[s1],d)
agent.get_delta(sess)
sess.run([agent.update_critic_weights,agent.update_actor_weights],feed_dict={agent.state_in:agent.time_step_info['s']})
agent.update_I()
s = s1
agent.save_model(path,sess)
except Exception as e:
print(e)

• I think this could be a good, on-topic question, but the inclusion of all the code is a distraction. I suggest give the high level hyperparameters, and just link the full code it in case someone feels like replicating your experiment. There might be a bug or misunderstanding in your implementation, but realistically no-one here is going to read through 100s of lines of your project code to help find it. However, you could still get an answer to the core question: How long should this experiment take to succeed, and is the specific agent type you have chosen suitable? – Neil Slater Sep 10 '18 at 15:04
• @NeilSlater Thank you, I will Edit the question following your suggestion. – Diego Orellana Sep 10 '18 at 15:06
• I would expect the actor to have 8 outputs representing $(\mu_0, \sigma_0, \mu_1, \sigma_1, \mu_2, \sigma_2, \mu_3, \sigma_3)$ for the 4 action dimensions, and the critic to have 1 output representing state value. Not clear if that is what you have implemented here (it would also be valid to have 5 outputs in the actor, sharing the std dev, or some other arrangement - looking at part of the method it seems you might be using a fixed std dev throughout, which may be valid theoretically but would need careful tuning). Can you explain the policy representation that your actor network is using? – Neil Slater Sep 10 '18 at 19:59
• It's very hard to find a basic "vanilla" Actor-Critic implementation, almost all the available sources are A3C implementations, which would not be a fair comparison. That makes this question hard to answer simply (by loading up Gym and trying some hyperparams with someone else's tested agent). I am looking into it though . . . – Neil Slater Sep 10 '18 at 20:12
• @NeilSlater Thank you for your interest. I think I'm close to solve it. What I'm doing is solving the cart-pole problem using the same code. I had a bug in the gradient calculation. I will post the code once is correct maybe somebody finds it useful. – Diego Orellana Sep 10 '18 at 21:34