2
$\begingroup$

I'm trying to solve the OpenAI BipedalWalker-v2 by using a one-step actor-critic agent. I'm implementing the solution using python and tensorflow.

I'm following this pseudo-code taken from the book Reinforcement Learning An Introduction by Richard S. Sutton and Andrew G. Barto.

enter image description here

in summary, my question can be reduced to the following:

  • Is it a good idea to implement a one-step actor-critic algorithm to solve the OpenAI BipedalWalker-v2 problem? If not what would be a good approach? If yes; how long would it take to converge?
  • I run the algorithm for 20000 episodes, each episode has an avg of 400 steps, for each step, I immediately update the weights. The results are not better than random. I have tried different standard deviations (for my normal distribution that represents pi), different NN sizes for the Critic and Actor, and different learning-steps for the optimizer algorithm. The results never improve. I don't know what I'm doing wrong.

My Agent Class

import tensorflow as tf
import numpy as np
import gym
import matplotlib.pyplot as plt

class agent_episodic_continuous_action():
    def __init__(self, lr,gamma,sample_variance, s_size,a_size,dist_type):
       ... #agent parameters

    def save_model(self,path,sess):    
    def load_model(self,path,sess):       
    def weights_init_actor(self,hidd_layer,mean,stddev): #to have control over the weights initialization      
    def weights_init_critic(self,hidd_layer,mean,stddev):  #to have control over the weights initialization            
    def create_actor_brain(self,hidd_layer,hidd_act_fn,output_act_fn,mean,stddev):  #actor is represented by a fully connected NN      
    def create_critic_brain(self,hidd_layer,hidd_act_fn,output_act_fn,mean,stddev): #critic is represented by a fully connected NN      
    def critic(self):            
    def get_delta(self,sess):                 
    def normal_dist_prob(self): #Actor pi distribution is a normal distribution whose mean comes from the NN 
    def create_actor_loss(self): 
    def create_critic_loss(self):
    def sample_action(self,sess,state): #Sample actions from the normal dist. Whose mean was aprox. By the NN
    def calculate_actor_loss_gradient(self):
    def calculate_critic_loss_gradient(self):   
    def update_actor_weights(self):
    def update_critic_weights(self):
    def update_I(self):  
    def reset_I(self):      
    def update_time_step_info(self,s,a,r,s1,d):  
    def create_graph_connections(self):
    def bound_actions(self,sess,state,lower_limit,uper_limit):  

Agent instantiation

tf.reset_default_graph()
agent= agent_episodic_continuous_action(learning-step=1e-3,gamma=0.99,pi_stddev=0.02,s_size=24,a_size=4,dist_type="normal")
agent.create_actor_brain(hidden_layers=[12,5],hidden_layers_fct="relu",output_layer="linear",mean=0.0,stddev=0.14)
agent.create_critic_brain(hidden_layers=[12,5],hidden_layers_fct="relu",output_layer="linear",mean=0.0,stddev=0.14)
agent.create_graph_connections()

path = "/home/diego/Desktop/Study/RL/projects/models/biped/model.ckt"   
env = gym.make('BipedalWalker-v2')
uper_action_limit = env.action_space.high
lower_action_limit = env.action_space.low   
total_returns=[]

Training loops

with tf.Session() as sess:
    try:
        sess.run(agent.init)
        sess.graph.finalize()
        #agent.load_model(path,sess)        
        for i in range(1000): 
            agent.reset_I()
            s = env.reset()    
            d = False
            while (not d):
                a=agent.bound_actions(sess,s,lower_action_limit,uper_action_limit)  
                s1,r,d,_ = env.step(a)
                #env.render()
                agent.update_time_step_info([s],[a],[r],[s1],d)                 
                agent.get_delta(sess)
                sess.run([agent.update_critic_weights,agent.update_actor_weights],feed_dict={agent.state_in:agent.time_step_info['s']})
                agent.update_I()  
                s = s1
        agent.save_model(path,sess)    
    except Exception as e:
        print(e)
$\endgroup$
  • 3
    $\begingroup$ I think this could be a good, on-topic question, but the inclusion of all the code is a distraction. I suggest give the high level hyperparameters, and just link the full code it in case someone feels like replicating your experiment. There might be a bug or misunderstanding in your implementation, but realistically no-one here is going to read through 100s of lines of your project code to help find it. However, you could still get an answer to the core question: How long should this experiment take to succeed, and is the specific agent type you have chosen suitable? $\endgroup$ – Neil Slater Sep 10 '18 at 15:04
  • 1
    $\begingroup$ @NeilSlater Thank you, I will Edit the question following your suggestion. $\endgroup$ – Diego Orellana Sep 10 '18 at 15:06
  • 2
    $\begingroup$ I would expect the actor to have 8 outputs representing $(\mu_0, \sigma_0, \mu_1, \sigma_1, \mu_2, \sigma_2, \mu_3, \sigma_3)$ for the 4 action dimensions, and the critic to have 1 output representing state value. Not clear if that is what you have implemented here (it would also be valid to have 5 outputs in the actor, sharing the std dev, or some other arrangement - looking at part of the method it seems you might be using a fixed std dev throughout, which may be valid theoretically but would need careful tuning). Can you explain the policy representation that your actor network is using? $\endgroup$ – Neil Slater Sep 10 '18 at 19:59
  • 1
    $\begingroup$ It's very hard to find a basic "vanilla" Actor-Critic implementation, almost all the available sources are A3C implementations, which would not be a fair comparison. That makes this question hard to answer simply (by loading up Gym and trying some hyperparams with someone else's tested agent). I am looking into it though . . . $\endgroup$ – Neil Slater Sep 10 '18 at 20:12
  • $\begingroup$ @NeilSlater Thank you for your interest. I think I'm close to solve it. What I'm doing is solving the cart-pole problem using the same code. I had a bug in the gradient calculation. I will post the code once is correct maybe somebody finds it useful. $\endgroup$ – Diego Orellana Sep 10 '18 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.