There is this nice result for policy gradients that the gradient of some performance measure, $\nabla v_{\pi_{\theta}}(s_0)$ (here, in the episodic case for the starting state $s_0$ and policy $\pi$, parametrised by some weights $\theta$) is equal to the expectation gradient of the logarithm of the policy, i.e.
$$\nabla v_{\pi_{\theta}}(s_0)=\mathbb{E}\Big{[}\sum_{t=0}^{T-1}\nabla_\theta\log(\pi_{\theta}(a_t|s_t)]\cdot G_t\Big{]},$$
where $G_t$ is the discounted future reward from state $s_t$ onward and $s_T$ the final state of some trajectory $(s_0, a_0, s_1, a_1, ..., s_{T-1}, a_{T-1}, s_T)$.
Now, when using a softmax policy, $\nabla_\theta\log(\pi_{\theta}(a_t|s_t)$ can be written as
$$\nabla_\theta\log(\pi_{\theta}(a_t|s_t))=\phi(s_t,a_t)-\mathbb{E}[\phi(s_t,\cdot)],$$
where $\phi(s,a)$ is some input vector of a state-action tuple.
However: what exactly is this vector? A typical input with policy gradients (for example in a neural network) is a feature vector for the state and the output a vector with dimensions equal to the number of actions, e.g. $(14, 15, 11, 17)^T$ for four possible actions. The softmax-function now scales these outputs, which results in the logits $(.042, .114, .002, .842)^T$ in this example.
What I would usually do in neural networks is take some input vector, for example something that describes if there are borders in a grid world, e.g. $\phi(s)=(1, 0, 0, 1)^T$, and multiply that with my weight matrix $\theta$ (and add biases b), i.e. $\theta\phi(s)+b$. So, continuing above example, $1\cdot \theta_{1,1} + 0\cdot \theta_{1,2} + 0\cdot \theta_{1,3} + 1\cdot \theta_{1,4} = 14$ and $1\cdot \theta_{2,1} + 0\cdot \theta_{2,2} + 0\cdot \theta_{2,3} + 1\cdot \theta_{2,4} = 15$.
But what is $\phi(s,a)$ here? And how would I compute $\nabla_\theta\log(\pi_{\theta}(a|s))=\phi(s,a)-\mathbb{E}[\phi(s,\cdot)]$?