46
votes
Accepted
What is the relation between Q-learning and policy gradients methods?
However, both approaches appear identical to me i.e. predicting the maximum reward for an action (Q-learning) is equivalent to predicting the probability of taking the action directly (PG).
Both ...
- 30.2k
16
votes
Accepted
What are the differences between Q-Learning and A*?
Q-learning and A* can both be viewed as search algorithms, but, apart from that, they are not very similar.
Q-learning is a reinforcement learning algorithm, i.e. an algorithm that attempts to find a ...
- 39.5k
14
votes
What is the relation between Q-learning and policy gradients methods?
This Tutorial by OpenAI offers a great comparison of different RL methods.
I'll try to summarize the differences between Q-Learning and Policy Gradient methods:
Objective Function
In Q-Learning we ...
- 451
14
votes
Accepted
What does the symbol $\mathbb E$ mean in these equations?
That's the Expected Value operator. Intuitively, it gives you the value that you would "expect" ("on average") the expression after it (often in square or other brackets) to have. Typically that ...
- 10.1k
13
votes
Accepted
Why does DQN require two different networks?
My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one.
It has no real impact on computation ...
- 30.2k
13
votes
Why doesn't Q-learning converge when using function approximation?
Here's an intuitive description answer:
Function approximation can be done with any parameterizable function. Consider the problem of a $Q(s,a)$ space where $s$ is the positive reals, $a$ is $0$ or $...
- 9,207
13
votes
What is the difference between Q-learning, Deep Q-learning and Deep Q-network?
Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN).
Tabular Q-learning (TQL)
Deep Q-learning (DQL)
...
- 39.5k
12
votes
Can Q-learning be used for continuous (state or action) spaces?
Q-learning for continuous state spaces
Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the ...
- 30.2k
10
votes
How do we prove the n-step return error reduction property?
Let's start by looking at:
$$\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert.$$
We can rewrite this by plugging in the definition of $G_{t:t+n}$:
\...
- 10.1k
10
votes
Accepted
What exactly is the advantage of double DQN over DQN?
In $Q$-learning there is what is known as a maximisation bias. That is because the update target is $r + \gamma \max_a Q(s,a)$. If you slightly overestimate your $Q$-value then this error gets ...
- 4,665
9
votes
Is Q-learning a type of model-based RL?
Tabular Q-Learning does not explicitly create a model of the transition function. It does not generate any output that you can afterwards use as a function to predict what the next state s' will be ...
- 10.1k
8
votes
Accepted
How should I model all available actions of a chess game in deep Q-learning?
To model chess as a Markov decision problem (MDP) you can refer to the AlphaZero paper (Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm). The exact details can ...
- 552
8
votes
Are Q-learning and SARSA the same when action selection is greedy?
If we write the pseudo-code for the SARSA algorithm we first initialise our hyper-parameters etc. and then initialise $S_t$, which we use to choose $A_t$ from our policy $\pi(a|s)$. Then for each $t$ ...
- 4,665
8
votes
Accepted
Is Q-learning only capable of learning a deterministic policy?
If we assume a tabular setting, then Q-learning converges to the optimal state-action value function, from which an optimal policy can be derived, provided a few conditions are met.
In finite MDPs, ...
- 39.5k
7
votes
Accepted
Is the discount not needed in a deterministic environment for Reinforcement Learning?
The motivation for adding the discount factor $\gamma$ is generally, at least initially, based simply in "theoretical convenience". Ideally, we'd like to define the "objective" of an RL agent as ...
- 10.1k
7
votes
Accepted
My DQN is stuck and can't see where the problem is
After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were ...
- 173
7
votes
Accepted
Can Q-learning be used in a POMDP?
The usual (as presented in Reinforcement Learning: An Introduction) $Q$-learning and SARSA algorithms use (and update) a function of a state $s$ and action $a$, $Q(s, a)$. These algorithms assume that ...
- 39.5k
7
votes
Why doesn't Q-learning converge when using function approximation?
As far as I'm aware, it is still somewhat of an open problem to get a really clear, formal understanding of exactly why / when we get a lack of convergence -- or, worse, sometimes a danger of ...
- 10.1k
7
votes
Accepted
Does AlphaZero use Q-Learning?
Note: you mentioned in the comments that you are reading the old, pre-print version of the paper describing AlphaZero on arXiv. My answer will be for the "official", peer-reviewed, more recent ...
- 10.1k
7
votes
What does the symbol $\mathbb E$ mean in these equations?
$\mathbb E$ is the symbol for the expectation (or expected value).
To fully understand the concept of expected value, you need to understand the concept of random variable. An example should help ...
- 39.5k
7
votes
Accepted
What are some online courses for deep reinforcement learning?
Let me first say that deep RL is just the combination of RL with deep learning. So, if you study RL and deep learning, then studying deep RL should be straightforward. For this reason, this answer ...
- 39.5k
7
votes
Accepted
Is there any good reference for double deep Q-learning?
If you're interested in the theory behind Double Q-learning (not deep!), the reference paper would be Double Q-learning by Hado van Hasselt (2010).
As for Double deep Q-learning (also called DDQN, ...
- 722
7
votes
Why do we need target network in deep Q learning?
In DQN that was presented in the original paper the update target for the Q-Network is $\left(r_t + \max_aQ(s_{t+1},a;\theta^-) - Q(s_t,a_t; \theta)\right)^2$ were $\theta^-$ is some old version of ...
- 4,665
7
votes
Accepted
What is the difference between Q-learning, Deep Q-learning and Deep Q-network?
In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, $Q(s, a)$. This is defined as
$$Q(s, a) = \mathbb{E}_\pi\left[ G_t | S_t = s, ...
- 4,665
7
votes
Deep Q-Learning "catastrophic drop" reasons?
This is a case of overfitting the Q function leading to compounding errors when selecting actions.
You have been training your policy for too long on the same data distribution.
Overfitting Q ...
- 532
6
votes
Accepted
Deep Q-Learning poor convergence on Stochastic Environment
The inputs that you describe seem like they should be sufficient for a DQN-based agent to learn a good strategy for playing Minesweeper, regardless of whether or not the starting layout changes. The ...
- 10.1k
6
votes
Accepted
How do updates in SARSA and Q-learning differ in code?
Picking actions and making updates should be treated as separate things. For Q-learning you also need to explore by using some exploration strategy (e.g. $\epsilon$-greedy).
Steps for Q-learning:
...
- 2,336
6
votes
What is the difference between the $\epsilon$-greedy and softmax policies?
The $\epsilon$-greedy policy is a policy that chooses the best action (i.e. the action associated with the highest value) with probability $1-\epsilon \in [0, 1]$ and a random action with probability $...
- 39.5k
6
votes
Accepted
Why do my rewards reduce after extensive training using D3QN?
It is not 100% clear, but this seems like an instance of catastrophic forgetting. This is something that often impacts reinforcement learning.
I have answered a very similar question on Data Science ...
- 30.2k
5
votes
How does Q-learning work in stochastic environments?
How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case?
The ...
- 10.1k
Only top scored, non community-wiki answers of a minimum length are eligible