When training a AI RL agent to play a game there'll be situations where the AI cannot perform certain actions lest they violate the game rules. That's easy to handle, and I can set illegal actions to some large negative amount so when doing an argmax they won't be selected. Or if I use softmax I can set probabilities of illegal actions to zero and then re-calculate softmax on the remaining legal states. Indeed, I believe this is what David Silver was referring to when asked this question at a presentation/lecture of AlphaZero:


But doing so changes the output from the network and surely changes things when performing the backprop once a reward is known.

How does one handle that?

Would I set the illegal actions to the mean of the legal actions, or zero...?

  • $\begingroup$ possible duplicate $\endgroup$ – Brale Mar 10 '19 at 23:11
  • $\begingroup$ Specifically, this has been thoroughly discussed in the comment chain you participated in under this answer. $\endgroup$ – Philip Raeisghasem Mar 11 '19 at 7:14
  • $\begingroup$ Please read my question carefully. This is not a duplicate and I've not forgotten about discussions I've participated in. I'm asking specifically about how the modified actions should be handled with the backprop process. i.e. not how to remove illegal actions, but what to set the illegal action values to when they're fed back in for backprop - hence the title "How to back-propagate illegal actions". $\endgroup$ – BigBadMe Mar 11 '19 at 7:47
  • 1
    $\begingroup$ in any case you would backprop only through the action that you chose, so do the same thing that you do with regular version of the algorithm that doesn't have illegal actions when you make the update. And when choosing the actions disregard the illegal actions $\endgroup$ – Brale Mar 11 '19 at 10:16
  • 1
    $\begingroup$ Sorry for misunderstanding. To expand upon @Brale_'s comment, because illegal actions are never taken, the agent never receives any feedback on them in the form of reward (or lack of reward). So it isn't even possible to backprop through illegal actions. $\endgroup$ – Philip Raeisghasem Mar 11 '19 at 11:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.