I have been trying to implement the ACER algorithm for continuous action spaces in reinforcement learning. The paper for the algorithm can be found here:
I have implemented parts of the algorithm, but I have encountered some roadblocks that I have not been able to figure out.
The following is the pseudo-code provided in the paper:
Here is what I have implemented so far:
states = tf.convert_to_tensor(trajectory.state) actions = tf.squeeze(tf.convert_to_tensor(trajectory.action), axis=-1) rewards = tf.convert_to_tensor(trajectory.reward) dones = tf.convert_to_tensor(trajectory.done) explore_means, state_values, action_values = actor_critic(states, actions) average_means, *_ = brain.average_actor_critic(states) k = len(trajectory.state) d = env.action_space.shape # Policies explore_policies = k*[None] behavior_policies = k*[None] average_policies = k*[None] # Tracking explore_actions = np.zeros([k, d]) importance_weights = np.zeros([k, 1]) explore_importance_weights = np.zeros([k, 1]) truncation_parameters = np.zeros([k, 1]) for i in range(k): behavior_policy = tfd.MultivariateNormalDiag( loc=trajectory.statistics[i], scale_diag=tf.ones(d)*POLICY_STD ) explore_policy = tfd.MultivariateNormalDiag( loc=explore_means[i], scale_diag=tf.ones(d)*POLICY_STD ) average_policy = tfd.MultivariateNormalDiag( loc=average_means[i], scale_diag=tf.ones(d)*POLICY_STD ) explore_action = explore_policy.sample() importance_weight = explore_policy.prob(actions[i]) / behavior_policy.prob(actions[i]) explore_importance_weight = explore_policy.prob(explore_action) / behavior_policy.prob(explore_action) truncation_parameter = min(1, (importance_weight)**d) behavior_policies[i] = behavior_policy explore_policies[i] = explore_policy average_policies[i] = average_policy explore_actions[i] = explore_action importance_weights[i] = importance_weight explore_importance_weights[i] = explore_importance_weight truncation_parameters[i] = truncation_parameter explore_actions = tf.convert_to_tensor(explore_actions, dtype=tf.float32) importance_weights = tf.convert_to_tensor(importance_weights, dtype=tf.float32) explore_importance_weights = tf.convert_to_tensor(explore_importance_weights, dtype=tf.float32) truncation_parameters = tf.convert_to_tensor(truncation_parameters, dtype=tf.float32) q_ret = values[-1] if not dones[-1] else tf.zeros(1) q_opc = tf.identity(q_ret) for i in reversed(range(k - 1)): q_ret = rewards[i] + GAMMA*q_ret q_opc = rewards[i] + GAMMA*q_opc # Compute quantities needed for trust region updating c = TRUNCATION_PARAMETER with tf.GradientTape(persistent=True) as tape: tape.watch(explore_policies[-2].loc) log_prob = explore_policies[-2].log_prob(actions[-2]) explore_log_prob = explore_policies[-2].log_prob(explore_actions[-2]) kl_div = tfp.distributions.kl_divergence(average_policies[-2], explore_policies[-2]) lp_grad = tape.gradient(log_prob, explore_policies[-2].loc) elp_grad = tape.gradient(explore_log_prob, explore_policies[-2].loc) kld_grad = tape.gradient(kl_div, explore_policies[-2].loc) term1 = min(c, importance_weights[-2])*lp_grad*(q_opc - state_values[-2]) term2 = tf.nn.relu(1 - (c / explore_importance_weights[-2]))*(action_values[-2] - state_values[-2])*elp_grad g = term1 + term2
So the goal here was to implement it exactly the way they have it in the paper and then afterwards optimize it for doing batches of trajectories. For now, however, it is sufficient for the purposes of learning.
My confusion comes from the use of differentials in this algorithm. I don't know what the specific type is for them, such as whether they are using it for the loss value to optimize on or if they are storing the gradients that will be used for updating. Another issue I am having is that it is not clear what they mean by this line:
I don't understand why they are using a partial derivative here if there is clearly more than one parameter in the neural network. Maybe they mean the gradient I am not sure, however.
So what would be helpful is if anybody has some guidance as to what they are getting at in this portion of the paper or if anybody has some advice as to what steps need to be taken in TensorFlow 2.0 to implement this algorithm.
Any help would be greatly appreciated! Thanks!!