2
$\begingroup$

If I trained a neural network with 4 outputs (one for each action: move down, up, left, and right) to move an agent through a grid (deterministic problem). The output of the neural network is a probability distribution over the 4 actions, due to the softmax activation function.

Is the policy (based on the neural network) a stochastic policy, even if the action space is discrete?

Improve this question
$\endgroup$
6
  • $\begingroup$ Yes, for aforementioned case the policy is stochastic. You can select the action for example based on $\epsilon$-greedy policy. which is also a stochastic policy. $\endgroup$
    – Swakshar Deb
    Feb 3, 2021 at 13:02
  • $\begingroup$ You said, "Is the policy (based in the neural network) a stochastic policy? even if the action space is discrete?" Why can't policy become stochastic in discrete action space? I can't understand your perspective. $\endgroup$
    – Swakshar Deb
    Feb 3, 2021 at 13:06
  • $\begingroup$ I just want to give more details about the problem when i mentioned the discrete action space. Sorry, it's not important. $\endgroup$
    – Xtalker
    Feb 3, 2021 at 14:03
  • $\begingroup$ In summary my question is: A policy Is deterministic because it return an action or return a probability"; or because the policy is applied in a stochastic enviroment or deterministic enviroment? $\endgroup$
    – Xtalker
    Feb 3, 2021 at 14:09
  • $\begingroup$ Hi @Xtailker. The policy being deterministic means that the agent would always pick the same action in the same state. The policy being stochastic means that the action that is picked determined by the probabilities. So, if after computing softmax for the state, you use a random number to determine which action to use, then your policy is stochastic. If, for example, you decided to always use Argmax for deciding your action - then your policy would be deterministic. However, if you add probability epsilon to pick a random action instead of argmax, your policy would again become stochastic. $\endgroup$
    – mark mark
    Feb 3, 2021 at 14:42

1 Answer 1

Reset to default
1
$\begingroup$

Is the policy (based in the neural network) a stochastic policy? even if the action space is discrete?

Yes. A discrete action space does not require a deterministic policy - it is possible to assign arbitrary probabilities to each action in each state provided each probability is in the range $[0,1]$ and the sum across all allowed actions is $1$. The two concepts of determinism and discrete actions are entirely separate.

The optimal policy in many situations can be deterministic. If there is only one deterministic optimal policy, your learned policy should be also close to deterministic if the learning process has been successful. That is, the probabilities of optimal actions should all be close to $1$, all the rest close to $0$.

If there is more than one possible optimal policy, your learning agent may have learned a stochastic "mix" of them where in some states it is equally good to take more than one action and the probabilities may be split between those good actions. This will still be optimal and not a problem. If that is the case you should expect to see many action choices close to $0$ and in each state a select few (maybe one) that sum to close to $1$ between them.

In the case of discrete actions, you can derive a deterministic policy from your neural network function by taking the argmax of the action probabilities. This is worth trying. It will round away bad actions that are close to 0 probability due to approximation in the neural network.

In practice sometimes a little randomness in a policy works better for real-world problems with imprecise measurements or other unknowns. It may even be necessary for adversarial environments or where there is key information missing. The only way to find out though is to try both stochastic and deterministic interpretations of the neural network output for your policy.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank's for your answer. But, is deterministic or stochastic because "the policy return an action or return a probability" o because the policy is applied in a enviroment stochastic or deterministic? $\endgroup$
    – Xtalker
    Feb 3, 2021 at 14:07
  • $\begingroup$ @Xtalker Sorry I do not understand your comment at all. Which components are you asking whether they are deterministic or stochastic? And in what cases? $\endgroup$
    – Neil Slater
    Feb 3, 2021 at 14:12
  • 1
    $\begingroup$ The policy is the probability distribution over the actions. If for any action, $a$, $P(a) = 1$, then this is deterministic policy otherwise the probability distribution over action is stochastic policy. Note that, don't confuse stochastic environment with stochastic policy. They are not the same thing. You can have a stochastic policy in a deterministic environment. $\endgroup$
    – Swakshar Deb
    Feb 3, 2021 at 14:21
  • $\begingroup$ Thinking the policy was trained in supervised mode, in a deterministic problem (such as a rubix cube). The action space is discrete and finite, and all actions has deterministics results. The output of the neural network will be a softmax function with a probability distribution about the confidence of the next action. However, the enviroment it is deterministic $\endgroup$
    – Xtalker
    Feb 3, 2021 at 14:23
  • $\begingroup$ @Xtalker In that case the policy produced natually by the method you have used is stochastic (although it could approximate a deterministic one). You may find that changing the softmax to an argmax will give you a deterministic policy suitable for the problem, as described in the answer. $\endgroup$
    – Neil Slater
    Feb 3, 2021 at 15:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .