# Implementing Actor-Critic with Experience Replay for Continuous Action Spaces

I have been trying to implement the ACER algorithm for continuous action spaces in reinforcement learning. The paper for the algorithm can be found here:

I have implemented parts of the algorithm, but I have encountered some roadblocks that I have not been able to figure out.

The following is the pseudo-code provided in the paper:

Here is what I have implemented so far:

states = tf.convert_to_tensor(trajectory.state)
actions = tf.squeeze(tf.convert_to_tensor(trajectory.action), axis=-1)
rewards = tf.convert_to_tensor(trajectory.reward)
dones = tf.convert_to_tensor(trajectory.done)

explore_means, state_values, action_values = actor_critic(states, actions)

average_means, *_ = brain.average_actor_critic(states)

k = len(trajectory.state)
d = env.action_space.shape[0]

# Policies
explore_policies = k*[None]
behavior_policies = k*[None]
average_policies = k*[None]

# Tracking
explore_actions = np.zeros([k, d])
importance_weights = np.zeros([k, 1])
explore_importance_weights = np.zeros([k, 1])
truncation_parameters = np.zeros([k, 1])

for i in range(k):

behavior_policy = tfd.MultivariateNormalDiag(
loc=trajectory.statistics[i],
scale_diag=tf.ones(d)*POLICY_STD
)

explore_policy = tfd.MultivariateNormalDiag(
loc=explore_means[i],
scale_diag=tf.ones(d)*POLICY_STD
)

average_policy = tfd.MultivariateNormalDiag(
loc=average_means[i],
scale_diag=tf.ones(d)*POLICY_STD
)

explore_action = explore_policy.sample()

importance_weight = explore_policy.prob(actions[i]) / behavior_policy.prob(actions[i])
explore_importance_weight = explore_policy.prob(explore_action) / behavior_policy.prob(explore_action)

truncation_parameter = min(1, (importance_weight)**d)

behavior_policies[i] = behavior_policy
explore_policies[i] = explore_policy
average_policies[i] = average_policy
explore_actions[i] = explore_action
importance_weights[i] = importance_weight
explore_importance_weights[i] = explore_importance_weight
truncation_parameters[i] = truncation_parameter

explore_actions = tf.convert_to_tensor(explore_actions, dtype=tf.float32)
importance_weights = tf.convert_to_tensor(importance_weights, dtype=tf.float32)
explore_importance_weights = tf.convert_to_tensor(explore_importance_weights, dtype=tf.float32)
truncation_parameters = tf.convert_to_tensor(truncation_parameters, dtype=tf.float32)

q_ret = values[-1] if not dones[-1] else tf.zeros(1)
q_opc = tf.identity(q_ret)

for i in reversed(range(k - 1)):

q_ret = rewards[i] + GAMMA*q_ret
q_opc = rewards[i] + GAMMA*q_opc

# Compute quantities needed for trust region updating
c = TRUNCATION_PARAMETER

tape.watch(explore_policies[-2].loc)

log_prob = explore_policies[-2].log_prob(actions[-2])
explore_log_prob = explore_policies[-2].log_prob(explore_actions[-2])

kl_div = tfp.distributions.kl_divergence(average_policies[-2], explore_policies[-2])

term1 = min(c, importance_weights[-2])*lp_grad*(q_opc - state_values[-2])
term2 = tf.nn.relu(1 - (c / explore_importance_weights[-2]))*(action_values[-2] - state_values[-2])*elp_grad

g = term1 + term2


So the goal here was to implement it exactly the way they have it in the paper and then afterwards optimize it for doing batches of trajectories. For now, however, it is sufficient for the purposes of learning.

My confusion comes from the use of differentials in this algorithm. I don't know what the specific type is for them, such as whether they are using it for the loss value to optimize on or if they are storing the gradients that will be used for updating. Another issue I am having is that it is not clear what they mean by this line:

I don't understand why they are using a partial derivative here if there is clearly more than one parameter in the neural network. Maybe they mean the gradient I am not sure, however.

So what would be helpful is if anybody has some guidance as to what they are getting at in this portion of the paper or if anybody has some advice as to what steps need to be taken in TensorFlow 2.0 to implement this algorithm.

Any help would be greatly appreciated! Thanks!!