Skip to main content
Share Your Experience: Take the 2024 Developer Survey
6 votes
Accepted

Should I be decaying the learning rate and the exploration rate in the same manner?

First of all, I'd say that there is a reason to give Learning Rate (LR) and Exploration Rate (ER) the same decay: they play at the same scale (the number of successive batches you'll train your model ...
16Aghnar's user avatar
  • 601
5 votes
Accepted

In Q-learning, wouldn't it be better to simply iterate through all possible states?

If your algorithm is executed multiple (or enough) times using an outer loop, it would converge to similar results as Q-learning would with $\gamma = 0$ (as you don't look what is the expected future ...
kirua's user avatar
  • 424
5 votes
Accepted

Why does Q-learning converge under 100% exploration rate?

Q-learning is guaranteed to converge (in the tabular case) under some mild conditions, one of which is that in the limit we visit each state-action tuple infinitely many times. If your random random ...
David's user avatar
  • 4,920
5 votes
Accepted

What is the relation between a policy which is the solution to a MDP and a policy like $\epsilon$-greedy?

for example, the "greedy policy" always chooses the action with the highest expected return, no matter which state we are in The "no matter which state we are in" there is generally not true; in ...
Dennis Soemers's user avatar
  • 10.3k
4 votes
Accepted

Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?

You can indeed use UCB in the RL setting. See e.g. section 38.5 Upper Confidence Bounds for Reinforcement Learning (page 521) of the book Bandit Algorithms by Csaba Szepesvari and Tor Lattimore for ...
nbro's user avatar
  • 40.8k
4 votes
Accepted

In DQN, is it possible to make some actions more likely?

For single-step Q learning, the behaviour policy can be any stochastic policy without any further adjustment to the update rules. You don't have to use $\epsilon$-greedy based on current Q function ...
Neil Slater's user avatar
  • 32.5k
4 votes
Accepted

Is there an advantage in decaying $\epsilon$ during Q-Learning?

Yes Q-learning benefits from decaying epsilon in at least two ways: Early exploration. It makes little sense to follow whatever policy is implied by the initialised network closely, and more will be ...
Neil Slater's user avatar
  • 32.5k
4 votes
Accepted

Which policy do I need to use in updating Q function?

I am going to stick with Q learning here to keep things simple. Most value-based reinforcement learning used for optimal control will have some statement similar to: Choose $a$ from $s$ using policy ...
Neil Slater's user avatar
  • 32.5k
3 votes
Accepted

Is it the high probability action that is always selected by the agent in REINFORCE algorithm?

You sample according to the probability distribution $\pi(a \mid s, \theta)$, so you do not always take the action with the highest probability (otherwise there would be no exploration but just ...
nbro's user avatar
  • 40.8k
3 votes
Accepted

Which policy has to be followed by a player while construction of its own Q-table?

I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA). Q learning is an off policy algorithm, meaning that the Q values can be ...
harwiltz's user avatar
  • 1,136
3 votes

Why aren't exploration techniques, such as UCB or Thompson sampling, used in full RL problems?

Many techniques for the exploration/exploitation dilemma that are inspired by multi-armed bandit problems, such as UCB1, assume that you can explicitly enumerate all state-action pairs; in fact, multi-...
Dennis Soemers's user avatar
  • 10.3k
2 votes

In Q-learning, wouldn't it be better to simply iterate through all possible states?

In short, yes, provided that you have a small number of states. In pretty much any real system, the number of states is much higher than you could ever hope to explore exhaustively in any reasonable ...
DrMcCleod's user avatar
  • 603
2 votes
Accepted

In addition to the reward function, which other functions do I need to implement Q-learning?

In addition to the RF [*], you also need to define an exploratory policy (an example is the $\epsilon$-greedy), which allows you to explore the environment and learn the state-action value function $\...
nbro's user avatar
  • 40.8k
1 vote
Accepted

Methods for sequential decision optimization problem with nonlinear bayesian reward function

One possible alternative approach is incorporating deep reinforcement learning (DRL) techniques. These techniques are sequential since they incorporate lookahead, and they are designed to attempt to ...
DeepQZero's user avatar
  • 1,399
1 vote

Metrics to compare the exploraation of RL Algorithms

For 2D (labyrinth-like) tasks the measure is usually percentage of the space that was covered. Another option is distance moved (in any direction). Once you have an objective measure for exploration, ...
maxy's user avatar
  • 233
1 vote

Is the case of a big state space, should we use a softmax exploration policy rather than $\epsilon$-greedy for Q-Learning?

In single-step Q learning, you can use almost any exploration policy that you like, provided it covers all choices eventually. Usually you want to focus around the target policy, because that is the ...
Neil Slater's user avatar
  • 32.5k
1 vote

Are the two policies in SARSA for choosing an action the same?

For learning, it doesn't matter much how you choose the first action before starting the main loop. That is because the agent doesn't need to learn about transitions to the first state of an episode. ...
Neil Slater's user avatar
  • 32.5k
1 vote
Accepted

Is it possible to apply a particular exploration policy for the on-policy RL agents?

In part it depends on the on-policy method you are using. In general you are not free to change the policy arbitrarily for on-policy policy gradient methods such as PPO or A3C. However, if you are ...
Neil Slater's user avatar
  • 32.5k
1 vote

How to reduce the number of episodes before the agent learns in this game?

TLDR: Simplify your agent. Context: As you've noticed, it's not a hard game and it does not require a complex policy. You'd need a single neuron to solve it perfectly: ...
Andre Goulart's user avatar
1 vote

Strategy for playing a board game with Minimax algorithm

I'm not familiar with your game so I can't tell you what a good heuristic woul be in your specific case, but I can give you some advice on how to look for a good heuristic function. As a rule of thumb,...
Zekko's user avatar
  • 26

Only top scored, non community-wiki answers of a minimum length are eligible