3

You can indeed use UCB in the RL setting. See e.g. section 38.5 Upper Confidence Bounds for Reinforcement Learning (page 521) of the book Bandit Algorithms by Csaba Szepesvari and Tor Lattimore for the details. However, compared to $\epsilon$-greedy (widely used in RL), UCB1 is more computationally expensive, given that, for each action, you need to ...


2

Many techniques for the exploration/exploitation dilemma that are inspired by multi-armed bandit problems, such as UCB1, assume that you can explicitly enumerate all state-action pairs; in fact, multi-armed bandit problems usually only have just one "state", and then this requirement turns into only requiring the ability to enumerate actions. In RL ...


2

The first thing to note here is that your results seem aligned with the results commonly found in the bandit literature. Second thing to note would be that the performance of bandit algorithms is usually measured in terms of regret. This is the difference between (i) the amount of rewards accumulated by an oracle policy having prior knowledge about the true ...


1

Neil Slater's answer is very nice, but I have a couple more suggestions: You can use entropy regularization. Basically, you modify your loss function to penalize low policy entropy (so less loss for more entropy) which should prevent your policy from becoming "too deterministic" too early. You can also try maximum-entropy methods, like SAC, which employ a ...


1

I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state. Yes, having a stochastic policy function is the main way that a lot of policy gradient methods achieve exploration, including REINFORCE, A2C, A3C. Is it ...


1

So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout? The only way I could see this being calculated is if you iterate over all possible dropout combinations, or as an approximation sample say 100 or 1000 actions with different dropout, to get a rough distribution. I don't think this is feasible for practical reasons (the agent ...


Only top voted, non community-wiki answers of a minimum length are eligible