2
$\begingroup$

Do neural networks compute the probability distribution for policy gradient methods. If so, how do they compute an infinite probability distribution? How do you represent a continuous action policy with a neural network?

$\endgroup$
  • 1
    $\begingroup$ Hi and welcome to this community! Can you be more specific: which probability distribution are you referring to? $\endgroup$ – nbro Sep 15 at 21:50
  • $\begingroup$ I'm not an expert and I don't know much about it, but for a continuous action space, don't stochastic policy gradients compute a probability for performing each action, so if there are infinite actions, then the model will have to compute an infinite amount of probabilities $\endgroup$ – Josh Goldman Sep 15 at 21:53
1
$\begingroup$

Do neural networks compute the probability distribution for policy gradient methods.

In short, yes. It does not have to be neural networks, any trainable parametric function approximator based on gradients will do. Neural networks are a common choice, as are linear function approximators using selected basis functions.

If so, how do they compute an infinite probability distribution?

For background, this is an issue for generating stochastic policies in continuous action spaces only. In discrete action spaces, it is usually possible to compute an arbitrary probability density function for the whole action space, and sample from it to model the policy. It is also possible to compute a deterministic policy simply enough in continuous spaces - the input is the current state and the output is the action to take. The issue then is that this does not allow an agent to learn through exploration of the environment. To do that requires a stochastic policy.

If you want to generate a stochastic policy in continuous action spaces, you could discretise the space and sample from that using e.g. softmax to generate the action probabilities. Or you could have the approximation function do something more indirect: Output the parameters of a probability distribution that can be sampled from.

How do you represent a continuous action policy with a neural network?

Typically by having state features as input and the parameters of a PDF that can be sampled as the output. For instance, the network could output mean $\mu$ and standard deviation $\sigma$ of a normal distribution for an action value, and the policy is given by $\pi(a|s) = a \sim \mathbb{N}(\mu, \sigma)$.

This distribution can be sampled (there are simple methods to generate a sample from a normal distribution), and returns from following this policy used as feedback to the neural network using the policy gradient theorem. Assuming that there is an optimal deterministic policy to be found, the neural network can learn over time to home in on a specific mean with low standard deviation.

In some cases, the standard deviation can be treated as a hyper-parameter, similar to $\epsilon$ in $\epsilon$-greedy action selection, and might be decayed over time. In that case, a neural network can just output the mean action.

It is also possible to learn a deterministic policy through off-policy learning, adding a noise function to support exploration. This is what Deep Deterministic Policy Gradient does.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.