45
votes
Accepted
What is the relation between Q-learning and policy gradients methods?
However, both approaches appear identical to me i.e. predicting the maximum reward for an action (Q-learning) is equivalent to predicting the probability of taking the action directly (PG).
Both ...
- 26.6k
16
votes
Accepted
What are the differences between Q-Learning and A*?
Q-learning and A* can both be viewed as search algorithms, but, apart from that, they are not very similar.
Q-learning is a reinforcement learning algorithm, i.e. an algorithm that attempts to find a ...
- 37.1k
14
votes
What is the relation between Q-learning and policy gradients methods?
This Tutorial by OpenAI offers a great comparison of different RL methods.
I'll try to summarize the differences between Q-Learning and Policy Gradient methods:
Objective Function
In Q-Learning we ...
- 451
13
votes
Accepted
Why does DQN require two different networks?
My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one.
It has no real impact on computation ...
- 26.6k
13
votes
Why doesn't Q-learning converge when using function approximation?
Here's an intuitive description answer:
Function approximation can be done with any parameterizable function. Consider the problem of a $Q(s,a)$ space where $s$ is the positive reals, $a$ is $0$ or $...
- 9,037
12
votes
Accepted
What does the symbol $\mathbb E$ mean in these equations?
That's the Expected Value operator. Intuitively, it gives you the value that you would "expect" ("on average") the expression after it (often in square or other brackets) to have. Typically that ...
- 9,804
12
votes
What is the difference between Q-learning, Deep Q-learning and Deep Q-network?
Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN).
Tabular Q-learning (TQL)
Deep Q-learning (DQL)
...
- 37.1k
10
votes
How do we prove the n-step return error reduction property?
Let's start by looking at:
$$\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert.$$
We can rewrite this by plugging in the definition of $G_{t:t+n}$:
\...
- 9,804
10
votes
Can Q-learning be used for continuous (state or action) spaces?
Q-learning for continuous state spaces
Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the ...
- 26.6k
9
votes
Accepted
What exactly is the advantage of double DQN over DQN?
In $Q$-learning there is what is known as a maximisation bias. That is because the update target is $r + \gamma \max_a Q(s,a)$. If you slightly overestimate your $Q$-value then this error gets ...
- 4,420
8
votes
Is Q-learning a type of model-based RL?
Tabular Q-Learning does not explicitly create a model of the transition function. It does not generate any output that you can afterwards use as a function to predict what the next state s' will be ...
- 9,804
8
votes
Are Q-learning and SARSA the same when action selection is greedy?
If we write the pseudo-code for the SARSA algorithm we first initialise our hyper-parameters etc. and then initialise $S_t$, which we use to choose $A_t$ from our policy $\pi(a|s)$. Then for each $t$ ...
- 4,420
8
votes
Accepted
Is Q-learning only capable of learning a deterministic policy?
If we assume a tabular setting, then Q-learning converges to the optimal state-action value function, from which an optimal policy can be derived, provided a few conditions are met.
In finite MDPs, ...
- 37.1k
7
votes
Accepted
How should I model all available actions of a chess game in deep Q-learning?
To model chess as a Markov decision problem (MDP) you can refer to the AlphaZero paper (Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm). The exact details can ...
- 532
7
votes
Accepted
Is the discount not needed in a deterministic environment for Reinforcement Learning?
The motivation for adding the discount factor $\gamma$ is generally, at least initially, based simply in "theoretical convenience". Ideally, we'd like to define the "objective" of an RL agent as ...
- 9,804
7
votes
Accepted
My DQN is stuck and can't see where the problem is
After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were ...
- 173
7
votes
Why doesn't Q-learning converge when using function approximation?
As far as I'm aware, it is still somewhat of an open problem to get a really clear, formal understanding of exactly why / when we get a lack of convergence -- or, worse, sometimes a danger of ...
- 9,804
7
votes
Accepted
Does AlphaZero use Q-Learning?
Note: you mentioned in the comments that you are reading the old, pre-print version of the paper describing AlphaZero on arXiv. My answer will be for the "official", peer-reviewed, more recent ...
- 9,804
7
votes
What does the symbol $\mathbb E$ mean in these equations?
$\mathbb E$ is the symbol for the expectation (or expected value).
To fully understand the concept of expected value, you need to understand the concept of random variable. An example should help ...
- 37.1k
7
votes
Accepted
What are some online courses for deep reinforcement learning?
Let me first say that deep RL is just the combination of RL with deep learning. So, if you study RL and deep learning, then studying deep RL should be straightforward. For this reason, this answer ...
- 37.1k
7
votes
Accepted
Is there any good reference for double deep Q-learning?
If you're interested in the theory behind Double Q-learning (not deep!), the reference paper would be Double Q-learning by Hado van Hasselt (2010).
As for Double deep Q-learning (also called DDQN, ...
- 722
7
votes
Accepted
What is the difference between Q-learning, Deep Q-learning and Deep Q-network?
In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, $Q(s, a)$. This is defined as
$$Q(s, a) = \mathbb{E}_\pi\left[ G_t | S_t = s, ...
- 4,420
7
votes
Deep Q-Learning "catastrophic drop" reasons?
This is a case of overfitting the Q function leading to compounding errors when selecting actions.
You have been training your policy for too long on the same data distribution.
Overfitting Q ...
- 532
6
votes
Accepted
Can Q-learning be used in a POMDP?
The usual (as presented in Reinforcement Learning: An Introduction) $Q$-learning and SARSA algorithms use (and update) a function of a state $s$ and action $a$, $Q(s, a)$. These algorithms assume that ...
- 37.1k
6
votes
Why do we need target network in deep Q learning?
In DQN that was presented in the original paper the update target for the Q-Network is $\left(r_t + \max_aQ(s_{t+1},a;\theta^-) - Q(s_t,a_t; \theta)\right)^2$ were $\theta^-$ is some old version of ...
- 4,420
6
votes
Accepted
Why do my rewards reduce after extensive training using D3QN?
It is not 100% clear, but this seems like an instance of catastrophic forgetting. This is something that often impacts reinforcement learning.
I have answered a very similar question on Data Science ...
- 26.6k
5
votes
How does Q-learning work in stochastic environments?
How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?
The ...
- 9,804
5
votes
Accepted
Should the exploration rate be reset after each trial in Q-learning?
The exploration rate, typically parameterized as epsilon / ε, can be changed on every trial. It depends on the complexity of the model and the goals.
The simplest thing to do is keep exploration rate ...
- 281
5
votes
Accepted
What are other ways of handling invalid actions in scenarios where all rewards are either 0 (best reward) or negative?
1) Is there any way to set the initial Q-values for the actions?
You can generally do this, but you cannot specify specific weights for specific actions in specific states. Not through the network ...
- 906
5
votes
Accepted
Why is the target $r + \gamma \max_{a'} Q(s', a'; \theta_i^-)$ in the loss function of the DQN architecture?
This is the problem that reinforcement learning (RL) is trying to solve: What is the best way to behave when we don’t know what the right action is and only have a scalar (the reward (r) is a scalar) ...
- 3,707
Only top scored, non community-wiki answers of a minimum length are eligible