Extend the loss function from the single action to the n-action case per time step

Question

My question concerns a side question (which was not answered) asked here: How can policy gradients be applied in the case of multiple continuous actions?

I am trying to implement a simple policy gradient algorithm for a discrete multi-action reinforcement learning task. To be more precise, there are three actuators. At every time step, each of the actuators can perform one of three possible actions.

Is it possible to adjust the loss function from the single action case per time step

$$L = \log(P(a_1)) A$$

to the n-action case per time step like so?

$$L = (\log(P(a_1)) + \log(P(a_2))+ \dots + \log(P(a_n))) A$$?

Answer 1

If you consider every action independent from the others, I assume the equation might be right. Consider the case with two actions. $\pi(a_1,a_2)$ is the joint probability of the two actions which becomes $\pi(a_1)\cdot \pi(a_2)$ in case $a_1$ and $a_2$ are independent. Then, $\log(\pi(a_1,a_2))$ will become $\log(\pi(a_1))+\log(\pi(a_2))$. I suppose this logic can be applied in case there are multiple actions. However, this should be considered that $\pi(a_i)$ should be calculated with its own mean and variance. so I assume the policy network will have 2n outputs if we have n actions. because each action requires it's own mean and variance.