4
votes
Accepted
Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?
You can indeed use UCB in the RL setting. See e.g. section 38.5 Upper Confidence Bounds for Reinforcement Learning (page 521) of the book Bandit Algorithms by Csaba Szepesvari and Tor Lattimore for ...
- 38.2k
3
votes
Accepted
Why is Thompson Sampling considered a part of Reinforcement Learning?
Thompson Sampling (TS) is used in the context of bandits, which is a special case of the RL problem.
You can also use TS for the full RL problem, but that can lead to inefficient exploration. To know ...
- 38.2k
3
votes
Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?
Many techniques for the exploration/exploitation dilemma that are inspired by multi-armed bandit problems, such as UCB1, assume that you can explicitly enumerate all state-action pairs; in fact, multi-...
- 9,824
3
votes
Why am I getting better performance with Thompson sampling than with UCB or $\epsilon$-greedy in a multi-armed bandit problem?
The first thing to note here is that your results seem aligned with the results commonly found in the bandit literature.
Second thing to note would be that the performance of bandit algorithms is ...
- 722
2
votes
Should I use exploration strategy in Policy Gradient algorithms?
Neil Slater's answer is very nice, but I have a couple more suggestions:
You can use entropy regularization. Basically, you modify your loss function to penalize low policy entropy (so less loss for ...
- 1,071
2
votes
Accepted
Should I use exploration strategy in Policy Gradient algorithms?
I believe that if I follow the policy (sample an action from the policy) I make use of exploration because each action has a certain probability so I will explore all actions for a given state.
Yes, ...
- 27k
1
vote
Accepted
How to compute the action probabilities with Thompson sampling in deep Q-learning?
So, how can I calculate $\mu(a)$ when using Thompson Sampling based on dropout?
The only way I could see this being calculated is if you iterate over all possible dropout combinations, or as an ...
- 27k
Only top scored, non community-wiki answers of a minimum length are eligible