What's the value of making the RL agent's output stochastic opposed to deterministic?

Question

I have a question about a reinforcement learning problem.

I'm training an agent to add or delete pixels in a [12 x 12] 2D space (going to be 3D in the future). Its action space consists of two discrete outputs: x[0-12] and y[0-12].

What would be the value of instead outputting a (continuous) probabilistic output representation, like the [12 x 12] space with each pixel as a probability, and sampling from it. E.g. a softmax function applied to 144 (12*12) output nodes.

My environment is deterministic itself: taking action 𝑎 in state 𝑠 always results in the same next state 𝑠′.

I understand that this may be more difficult to train since the output space becomes continuous instead of discrete, and therefore bigger, but does stochastic/probabilistic output have any benefits over 1 discrete output?

Thanks!

Answer 1

My understanding of your question is, you have 2 designs:

1. A deterministic policy that outputs 2 scalar for x and y respectively.
2. A value function that outputs the probability of each pixel in the 2D grid.

If you choose the max of softmax on (2.), you'll get the same deterministic policy as (1.), assuming there are some tie-breaking designs. So I don't think choosing the max of softmax is a great policy.

So, the question becomes "deterministic policy vs. value function". Since the value function have more information, we can have special designs on exploration.

In (1.), if you use $\epsilon$-greedy on the discrete output, It'll take a random move with $\epsilon$ probability. When exploring (instead of exploiting), all actions except the best action will have the same probability.

However, in (2.), the probability can act like a value function (the value of $Q(s, a) \forall a$ in a certain state $s$), and you can use something like upper confidence bound (UCB) instead of $\epsilon$-greedy. If you are using neural networks to output this probability distribution, you can add an entropy term in the loss function like in A3C to encourage exploration.