I would recommend not trying to think of this in relation to supervised learning.
The policy $\pi(\cdot; \theta)$ is simply a function that is parameterised by theta. If we take a $\log$ of this function, it is still just a function. We want to take the (partial) derivative(s) of this function with respect to the parameters so that we can perform a gradient ascent step on the parameters.
A simple example can be shown by letting $\pi(a; \alpha, \beta) = \exp(\alpha + \beta a)$. In the policy gradient theorem we must first take a log of the policy which would give us $\log(\pi(a; \alpha, \beta)) = \alpha + \beta a$, and the partial derivatives wrt to the parameters are $\nabla_\alpha \log(\pi(a; \alpha, \beta)) = 1$ and $\nabla_\beta \log(\pi(a; \alpha, \beta)) = a$. We can then use these partial derivatives to perform a gradient ascent update in the direction of the gradient of our objective (the value function, which is of course what we want to maximise) for $\alpha$ and $\beta$ for a given return $G_t$ and action $a_t$ by
\alpha' = \alpha + G_t \times \nabla_\alpha \log(\pi(a_t; \alpha, \beta)) = \alpha + G_t \times 1 \; \\
\beta' = \beta + G_t \times \nabla_\alpha \log(\pi(a_t; \alpha, \beta)) = \beta + G_t a_t\;.
In practice, however, you're likely to need a much more complex function for the policy, typically a neural network of some description. However, everything translates to these more complex functions, you're just going to have many more partial derivatives to calculate.