# Exploration for softmax should be binary or continuous softmax?

Maybe it's silly to ask but for random exploration in an RL for choosing discrete action, that in the neural network last layer softmax will be used, what random samples should we provide? binary like (0,0,1,0,0,0) or continuous softmax like (0.1, 0.15, 0.45, 0.25, 0,5, 0.1)??

if the answer is continuous, what algorithm do you suggest? like generating random numbers between 0 and 1 and then using softmax? (this algorithm mostly provides close numbers and I think it's not the correct way)

• This post is unclear to me. What do you mean by "what random samples should we provide?" Provide to what? Moreover, what "random samples" are you talking about? Samples of what?
– nbro
May 3 '21 at 12:38
• Random samples of possible actions for exploration. In the exploitation, the neural network will provide the action for us but in the exploration, we have to generate these samples of action ourselves and feed them to the algorithm. The actual question is for a neural network with softmax in output, these random actions should be binary or continuous. May 3 '21 at 16:28
• If by samples you mean "actions" (which is what you just said), then I don't understand the question "actions should be continuous or binary".
– nbro
May 3 '21 at 17:23

1. One way to measure model (or epistemic) uncertainty is to use a Bayesian Network estimating the whole distribution $$p(\theta | D)$$, where \theta are your parameters and D is the collected dataset of experience. The entropy of $$p(\theta | D)$$ tells us the model uncertainty. Why Bayesian Network? Intuitively, if our estimator says all θs are equally likely given your data, then it means that you have no idea of what the model really is. On the other hand, if it says there is one only θ, that can possibly explain D, then you have high confidence in the model.
2. Another way to measure epistemic uncertainty is to use bootstrap ensembling. Instead of estimating the uncertainty of every single parameter in the Bayesian Net, learn N different networks! Use bootstrapping in order to generate N datasets (of same size as D) by resampling with replacement. Then train each network on a corresponding generated dataset. Then in testing, you can either sample uniformly a network ($$p(\theta | D) = \frac{1}{N}\sum_i^N{\delta({\theta}_i)}$$) or just average all networks predictions.