3
$\begingroup$

My question concerns a side question (which was not answered) asked here: How can policy gradients be applied in the case of multiple continuous actions?

I am trying to implement a simple policy gradient algorithm for a discrete multi-action reinforcement learning task. To be more precise, there are three actuators. At every time step, each of the actuators can perform one of three possible actions.

Is it possible to adjust the loss function from the single action case per time step

$$L = \log(P(a_1)) A$$

to the n-action case per time step like so?

$$L = (\log(P(a_1)) + \log(P(a_2))+ \dots + \log(P(a_n))) A$$?

$\endgroup$
1
  • $\begingroup$ Welcome to AI! Side question becoming a question in it's own right is perfectly acceptable. Thanks for raising it. $\endgroup$
    – DukeZhou
    Apr 26 '18 at 16:02
0
$\begingroup$

If you consider every action independent from the others, I assume the equation might be right. Consider the case with two actions. $\pi(a_1,a_2)$ is the joint probability of the two actions which becomes $\pi(a_1)\cdot \pi(a_2)$ in case $a_1$ and $a_2$ are independent. Then, $\log(\pi(a_1,a_2))$ will become $\log(\pi(a_1))+\log(\pi(a_2))$. I suppose this logic can be applied in case there are multiple actions. However, this should be considered that $\pi(a_i)$ should be calculated with its own mean and variance. so I assume the policy network will have 2n outputs if we have n actions. because each action requires it's own mean and variance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.