Skip to main content
Share Your Experience: Take the 2024 Developer Survey
47 votes
Accepted

What is the relation between Q-learning and policy gradients methods?

However, both approaches appear identical to me i.e. predicting the maximum reward for an action (Q-learning) is equivalent to predicting the probability of taking the action directly (PG). Both ...
Neil Slater's user avatar
  • 32.4k
16 votes
Accepted

What are the differences between Q-Learning and A*?

Q-learning and A* can both be viewed as search algorithms, but, apart from that, they are not very similar. Q-learning is a reinforcement learning algorithm, i.e. an algorithm that attempts to find a ...
nbro's user avatar
  • 40.8k
14 votes

What is the relation between Q-learning and policy gradients methods?

This Tutorial by OpenAI offers a great comparison of different RL methods. I'll try to summarize the differences between Q-Learning and Policy Gradient methods: Objective Function In Q-Learning we ...
Ray Walker's user avatar
14 votes

Why doesn't Q-learning converge when using function approximation?

Here's an intuitive description answer: Function approximation can be done with any parameterizable function. Consider the problem of a $Q(s,a)$ space where $s$ is the positive reals, $a$ is $0$ or $...
John Doucette's user avatar
14 votes
Accepted

What does the symbol $\mathbb E$ mean in these equations?

That's the Expected Value operator. Intuitively, it gives you the value that you would "expect" ("on average") the expression after it (often in square or other brackets) to have. Typically that ...
Dennis Soemers's user avatar
  • 10.3k
13 votes
Accepted

Why does DQN require two different networks?

My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one. It has no real impact on computation ...
Neil Slater's user avatar
  • 32.4k
13 votes

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN). Tabular Q-learning (TQL) Deep Q-learning (DQL) ...
nbro's user avatar
  • 40.8k
12 votes

Can Q-learning be used for continuous (state or action) spaces?

Q-learning for continuous state spaces Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the ...
Neil Slater's user avatar
  • 32.4k
11 votes
Accepted

What exactly is the advantage of double DQN over DQN?

In $Q$-learning there is what is known as a maximisation bias. That is because the update target is $r + \gamma \max_a Q(s,a)$. If you slightly overestimate your $Q$-value then this error gets ...
David's user avatar
  • 4,910
10 votes

How do we prove the n-step return error reduction property?

Let's start by looking at: $$\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert.$$ We can rewrite this by plugging in the definition of $G_{t:t+n}$: \...
Dennis Soemers's user avatar
  • 10.3k
9 votes

Is Q-learning a type of model-based RL?

Tabular Q-Learning does not explicitly create a model of the transition function. It does not generate any output that you can afterwards use as a function to predict what the next state s' will be ...
Dennis Soemers's user avatar
  • 10.3k
9 votes

Why doesn't Q-learning converge when using function approximation?

As far as I'm aware, it is still somewhat of an open problem to get a really clear, formal understanding of exactly why / when we get a lack of convergence -- or, worse, sometimes a danger of ...
Dennis Soemers's user avatar
  • 10.3k
8 votes
Accepted

How should I model all available actions of a chess game in deep Q-learning?

To model chess as a Markov decision problem (MDP) you can refer to the AlphaZero paper (Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm). The exact details can ...
Hai Nguyen's user avatar
8 votes

Are Q-learning and SARSA the same when action selection is greedy?

If we write the pseudo-code for the SARSA algorithm we first initialise our hyper-parameters etc. and then initialise $S_t$, which we use to choose $A_t$ from our policy $\pi(a|s)$. Then for each $t$ ...
David's user avatar
  • 4,910
8 votes

Deep Q-Learning "catastrophic drop" reasons?

This is a case of overfitting the Q function leading to compounding errors when selecting actions. You have been training your policy for too long on the same data distribution. Overfitting Q ...
devidduma's user avatar
  • 552
8 votes
Accepted

Is Q-learning only capable of learning a deterministic policy?

If we assume a tabular setting, then Q-learning converges to the optimal state-action value function, from which an optimal policy can be derived, provided a few conditions are met. In finite MDPs, ...
nbro's user avatar
  • 40.8k
7 votes
Accepted

Is the discount not needed in a deterministic environment for Reinforcement Learning?

The motivation for adding the discount factor $\gamma$ is generally, at least initially, based simply in "theoretical convenience". Ideally, we'd like to define the "objective" of an RL agent as ...
Dennis Soemers's user avatar
  • 10.3k
7 votes
Accepted

My DQN is stuck and can't see where the problem is

After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were ...
JCP's user avatar
  • 173
7 votes
Accepted

Can Q-learning be used in a POMDP?

The usual (as presented in Reinforcement Learning: An Introduction) $Q$-learning and SARSA algorithms use (and update) a function of a state $s$ and action $a$, $Q(s, a)$. These algorithms assume that ...
nbro's user avatar
  • 40.8k
7 votes
Accepted

Does AlphaZero use Q-Learning?

Note: you mentioned in the comments that you are reading the old, pre-print version of the paper describing AlphaZero on arXiv. My answer will be for the "official", peer-reviewed, more recent ...
Dennis Soemers's user avatar
  • 10.3k
7 votes

What does the symbol $\mathbb E$ mean in these equations?

$\mathbb E$ is the symbol for the expectation (or expected value). To fully understand the concept of expected value, you need to understand the concept of random variable. An example should help ...
nbro's user avatar
  • 40.8k
7 votes

What is the difference between the $\epsilon$-greedy and softmax policies?

The $\epsilon$-greedy policy is a policy that chooses the best action (i.e. the action associated with the highest value) with probability $1-\epsilon \in [0, 1]$ and a random action with probability $...
nbro's user avatar
  • 40.8k
7 votes
Accepted

What are some online courses for deep reinforcement learning?

Let me first say that deep RL is just the combination of RL with deep learning. So, if you study RL and deep learning, then studying deep RL should be straightforward. For this reason, this answer ...
nbro's user avatar
  • 40.8k
7 votes
Accepted

Is there any good reference for double deep Q-learning?

If you're interested in the theory behind Double Q-learning (not deep!), the reference paper would be Double Q-learning by Hado van Hasselt (2010). As for Double deep Q-learning (also called DDQN, ...
user5093249's user avatar
7 votes

Why do we need target network in deep Q learning?

In DQN that was presented in the original paper the update target for the Q-Network is $\left(r_t + \max_aQ(s_{t+1},a;\theta^-) - Q(s_t,a_t; \theta)\right)^2$ were $\theta^-$ is some old version of ...
David's user avatar
  • 4,910
7 votes
Accepted

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, $Q(s, a)$. This is defined as $$Q(s, a) = \mathbb{E}_\pi\left[ G_t | S_t = s, ...
David's user avatar
  • 4,910
7 votes

Does increasing the number of Q functions in Q-Learning scale?

Probably adding more Q estimators that are trained on separate data would not improve performance, and may even degrade it. At least there is no theoretical justification. Double Q learning addresses ...
Neil Slater's user avatar
  • 32.4k
7 votes
Accepted

Does increasing the number of Q functions in Q-Learning scale?

Yes, there are variations of Q-learning which use $n$ Q-functions named "enseble Q-learning" or "ensemble Q-functions". You can have a look at REDQ algorithm. The main benefit of ...
Luca Anzalone's user avatar
6 votes
Accepted

Why is the target $r + \gamma \max_{a'} Q(s', a'; \theta_i^-)$ in the loss function of the DQN architecture?

This is the problem that reinforcement learning (RL) is trying to solve: What is the best way to behave when we don’t know what the right action is and only have a scalar (the reward (r) is a scalar) ...
Jaden Travnik's user avatar
6 votes

How does Q-learning work in stochastic environments?

How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case? The ...
Dennis Soemers's user avatar
  • 10.3k

Only top scored, non community-wiki answers of a minimum length are eligible