How do I correctly apply action masking during DDPG training in Python?

Question

I'm implementing the Deep Deterministic Policy Gradient (DDPG) algorithm in PyTorch, and I'm facing issues with applying an action mask during the training process.

Currently, I apply an action mask in the simulation step to ensure only valid actions are selected. However, I'm uncertain whether I should also apply a mask during the training step. Specifically, I need help with:

Generating next_actions: Should I apply the action mask when generating next_actions using self.actor_target(next_states) for computing the Q-value target?

next_actions = self.actor_target(next_states)  # Should an action mask be applied here?

Calculating policy_loss: When calculating the policy loss using the main actor (self.actor), should I apply the action mask to self.actor(states) before passing it to the critic?

policy_loss = -self.critic(states, self.actor(states)).mean()  # Should an action mask be applied here?

I've tried both approaches (with and without masking), but the results are inconsistent, and I'm not sure which approach aligns with best practices in DDPG implementation. What is the recommended practice for handling action masks during DDPG training in this context?

Answer 1

During each DDPG mini-batch training step there're two places you need to apply the action mask.

When the next actions in a mini-batch are generated via the target actor as your self.actor_target(next_states) for computing the Q-value of the target critic, mask is required since it ensures that the target critic is only evaluating valid actions for computing the main critic's loss to be backpropagated later.

When the main actor generates actions at the current states in a mini-batch as your self.actor(states) to be added with OU noise schedule for exploration during training, mask is also required since it ensures that the main actor doesn’t propose invalid actions to update the policy. If the mask invalidates the action, you need to modify or resample a valid action.

However, you do not need to apply the mask during the policy_loss calculation since this loss is based on the main actor's actions and the main critic's evaluation where the two previously masked actions have been already applied.