This is a question I posted here. I am asking it on this StackExchange branch as well, so that more people who could potentially answer get to see the question.

In the A3C algorithm from the original paper:

the gradient with respect to log policy involves the term

$$\log \pi(a_i|s_i;\theta')$$

where $$s_i$$ is the state of the environment at time step $$i$$, and $$a_i$$ is the action produced by the policy. If I understand correctly, the output of the policy is a softmax function, so that if there are $$n$$ different actions, then we get the $$n$$-dimensional vector output

$$\pi(s_i;\theta')=\left(\frac{e^{o_1(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}},\frac{e^{o_2(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}},...,\frac{e^{o_n(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}}\right),$$

where the $$o_j(s)$$ are softmax layer activations obtained from forward propagation of state $$s_i$$ through the neural network.

Do I understand correctly that in the A3C algorithm above the term $$\log \pi(a_i|s_i;\theta')$$ refers to

$$\log \pi(a_i|s_i;\theta') = \log\left(\frac{e^{o_j(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}}\right)$$

with index $$j$$ referring to the position of the largest element in vector $$\pi(s_i;\theta')$$ above? Or maybe all action options should be contributing according to their probabilistic weights, like so

$$\log \pi(a_i|s_i;\theta') = \sum_{j=1}^n\log\left(\frac{e^{o_j(s_i)}}{\sum_{l=1}^n e^{o_l(s_i)}}\right)~~~?$$

Or perhaps neither of these expressions is correct? In that case, what is the correct explicit formula for the expression $$\log \pi(a_i|s_i;\theta')$$ in terms of softmax layer activations $$o_j$$?