6
votes
Accepted
Should I be decaying the learning rate and the exploration rate in the same manner?
First of all, I'd say that there is a reason to give Learning Rate (LR) and Exploration Rate (ER) the same decay: they play at the same scale (the number of successive batches you'll train your model ...
- 601
5
votes
Accepted
In Q-learning, wouldn't it be better to simply iterate through all possible states?
If your algorithm is executed multiple (or enough) times using an outer loop, it would converge to similar results as Q-learning would with $\gamma = 0$ (as you don't look what is the expected future ...
- 424
5
votes
Accepted
Why does Q-learning converge under 100% exploration rate?
Q-learning is guaranteed to converge (in the tabular case) under some mild conditions, one of which is that in the limit we visit each state-action tuple infinitely many times. If your random random ...
- 4,675
4
votes
Accepted
What is the relation between a policy which is the solution to a MDP and a policy like $\epsilon$-greedy?
for example, the "greedy policy" always chooses the action with the highest expected return, no matter which state we are in
The "no matter which state we are in" there is generally not true; in ...
- 10.1k
4
votes
Accepted
Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?
You can indeed use UCB in the RL setting. See e.g. section 38.5 Upper Confidence Bounds for Reinforcement Learning (page 521) of the book Bandit Algorithms by Csaba Szepesvari and Tor Lattimore for ...
- 39.6k
4
votes
Accepted
In DQN, is it possible to make some actions more likely?
For single-step Q learning, the behaviour policy can be any stochastic policy without any further adjustment to the update rules.
You don't have to use $\epsilon$-greedy based on current Q function ...
- 30.4k
4
votes
Accepted
Which policy do I need to use in updating Q function?
I am going to stick with Q learning here to keep things simple. Most value-based reinforcement learning used for optimal control will have some statement similar to:
Choose $a$ from $s$ using policy ...
- 30.4k
3
votes
Accepted
Is it the high probability action that is always selected by the agent in REINFORCE algorithm?
You sample according to the probability distribution $\pi(a \mid s, \theta)$, so you do not always take the action with the highest probability (otherwise there would be no exploration but just ...
- 39.6k
3
votes
Accepted
Which policy has to be followed by a player while construction of its own Q-table?
I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA).
Q learning is an off policy algorithm, meaning that the Q values can be ...
- 1,126
3
votes
Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?
Many techniques for the exploration/exploitation dilemma that are inspired by multi-armed bandit problems, such as UCB1, assume that you can explicitly enumerate all state-action pairs; in fact, multi-...
- 10.1k
3
votes
Accepted
Is there an advantage in decaying $\epsilon$ during Q-Learning?
Yes Q-learning benefits from decaying epsilon in at least two ways:
Early exploration. It makes little sense to follow whatever policy is implied by the initialised network closely, and more will be ...
- 30.4k
2
votes
In Q-learning, wouldn't it be better to simply iterate through all possible states?
In short, yes, provided that you have a small number of states.
In pretty much any real system, the number of states is much higher than you could ever hope to explore exhaustively in any reasonable ...
- 593
2
votes
Accepted
In addition to the reward function, which other functions do I need to implement Q-learning?
In addition to the RF [*], you also need to define an exploratory policy (an example is the $\epsilon$-greedy), which allows you to explore the environment and learn the state-action value function $\...
- 39.6k
1
vote
Accepted
Methods for sequential decision optimization problem with nonlinear bayesian reward function
One possible alternative approach is incorporating deep reinforcement learning (DRL) techniques. These techniques are sequential since they incorporate lookahead, and they are designed to attempt to ...
- 1,332
1
vote
Metrics to compare the exploraation of RL Algorithms
For 2D (labyrinth-like) tasks the measure is usually percentage of the space that was covered. Another option is distance moved (in any direction).
Once you have an objective measure for exploration, ...
- 223
1
vote
Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?
In single-step Q learning, you can use almost any exploration policy that you like, provided it covers all choices eventually. Usually you want to focus around the target policy, because that is the ...
- 30.4k
1
vote
Are the two policies in SARSA for choosing an action the same?
For learning, it doesn't matter much how you choose the first action before starting the main loop. That is because the agent doesn't need to learn about transitions to the first state of an episode.
...
- 30.4k
1
vote
Accepted
Is it possible to apply a particular exploration policy for the on-policy RL agents?
In part it depends on the on-policy method you are using. In general you are not free to change the policy arbitrarily for on-policy policy gradient methods such as PPO or A3C.
However, if you are ...
- 30.4k
1
vote
How to reduce the number of episodes before the agent learns in this game?
TLDR: Simplify your agent.
Context:
As you've noticed, it's not a hard game and it does not require a complex policy.
You'd need a single neuron to solve it perfectly: ...
- 854
1
vote
Strategy for playing a board game with Minimax algorithm
I'm not familiar with your game so I can't tell you what a good heuristic woul be in your specific case, but I can give you some advice on how to look for a good heuristic function.
As a rule of thumb,...
- 26
Only top scored, non community-wiki answers of a minimum length are eligible