1
$\begingroup$

I have been trying to implement the ACER algorithm for continuous action spaces in reinforcement learning. The paper for the algorithm can be found here:

I have implemented parts of the algorithm, but I have encountered some roadblocks that I have not been able to figure out.

The following is the pseudo-code provided in the paper:

Algorithm from Paper

Here is what I have implemented so far:

states = tf.convert_to_tensor(trajectory.state)
actions = tf.squeeze(tf.convert_to_tensor(trajectory.action), axis=-1)
rewards = tf.convert_to_tensor(trajectory.reward)
dones = tf.convert_to_tensor(trajectory.done)

explore_means, state_values, action_values = actor_critic(states, actions)

average_means, *_ = brain.average_actor_critic(states)

k = len(trajectory.state)
d = env.action_space.shape[0]

# Policies
explore_policies = k*[None]
behavior_policies = k*[None]
average_policies = k*[None]

# Tracking
explore_actions = np.zeros([k, d])
importance_weights = np.zeros([k, 1])
explore_importance_weights = np.zeros([k, 1])
truncation_parameters = np.zeros([k, 1])

for i in range(k):

    behavior_policy = tfd.MultivariateNormalDiag(
        loc=trajectory.statistics[i],
        scale_diag=tf.ones(d)*POLICY_STD
    ) 

    explore_policy = tfd.MultivariateNormalDiag(
        loc=explore_means[i],
        scale_diag=tf.ones(d)*POLICY_STD
    ) 

    average_policy = tfd.MultivariateNormalDiag(
        loc=average_means[i],
        scale_diag=tf.ones(d)*POLICY_STD
    )

    explore_action = explore_policy.sample()

    importance_weight = explore_policy.prob(actions[i]) / behavior_policy.prob(actions[i])
    explore_importance_weight = explore_policy.prob(explore_action) / behavior_policy.prob(explore_action)

    truncation_parameter = min(1, (importance_weight)**d)


    behavior_policies[i] = behavior_policy
    explore_policies[i] = explore_policy
    average_policies[i] = average_policy
    explore_actions[i] = explore_action
    importance_weights[i] = importance_weight
    explore_importance_weights[i] = explore_importance_weight
    truncation_parameters[i] = truncation_parameter


explore_actions = tf.convert_to_tensor(explore_actions, dtype=tf.float32)
importance_weights = tf.convert_to_tensor(importance_weights, dtype=tf.float32)
explore_importance_weights = tf.convert_to_tensor(explore_importance_weights, dtype=tf.float32)
truncation_parameters = tf.convert_to_tensor(truncation_parameters, dtype=tf.float32)


q_ret = values[-1] if not dones[-1] else tf.zeros(1)
q_opc = tf.identity(q_ret)

for i in reversed(range(k - 1)):

    q_ret = rewards[i] + GAMMA*q_ret
    q_opc = rewards[i] + GAMMA*q_opc


    # Compute quantities needed for trust region updating
    c = TRUNCATION_PARAMETER

    with tf.GradientTape(persistent=True) as tape:

        tape.watch(explore_policies[-2].loc)

        log_prob = explore_policies[-2].log_prob(actions[-2])
        explore_log_prob = explore_policies[-2].log_prob(explore_actions[-2])

        kl_div = tfp.distributions.kl_divergence(average_policies[-2], explore_policies[-2])


    lp_grad = tape.gradient(log_prob, explore_policies[-2].loc)    
    elp_grad = tape.gradient(explore_log_prob, explore_policies[-2].loc) 
    kld_grad = tape.gradient(kl_div, explore_policies[-2].loc) 


    term1 = min(c, importance_weights[-2])*lp_grad*(q_opc - state_values[-2])
    term2 = tf.nn.relu(1 - (c / explore_importance_weights[-2]))*(action_values[-2] - state_values[-2])*elp_grad

    g = term1 + term2

So the goal here was to implement it exactly the way they have it in the paper and then afterwards optimize it for doing batches of trajectories. For now, however, it is sufficient for the purposes of learning.

My confusion comes from the use of differentials in this algorithm. I don't know what the specific type is for them, such as whether they are using it for the loss value to optimize on or if they are storing the gradients that will be used for updating. Another issue I am having is that it is not clear what they mean by this line: enter image description here

I don't understand why they are using a partial derivative here if there is clearly more than one parameter in the neural network. Maybe they mean the gradient I am not sure, however.

So what would be helpful is if anybody has some guidance as to what they are getting at in this portion of the paper or if anybody has some advice as to what steps need to be taken in TensorFlow 2.0 to implement this algorithm.

Any help would be greatly appreciated! Thanks!!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.