46 votes
Accepted

What is the relation between Q-learning and policy gradients methods?

However, both approaches appear identical to me i.e. predicting the maximum reward for an action (Q-learning) is equivalent to predicting the probability of taking the action directly (PG). Both ...
Neil Slater's user avatar
  • 30.2k
16 votes
Accepted

What are the differences between Q-Learning and A*?

Q-learning and A* can both be viewed as search algorithms, but, apart from that, they are not very similar. Q-learning is a reinforcement learning algorithm, i.e. an algorithm that attempts to find a ...
nbro's user avatar
  • 39.5k
14 votes

What is the relation between Q-learning and policy gradients methods?

This Tutorial by OpenAI offers a great comparison of different RL methods. I'll try to summarize the differences between Q-Learning and Policy Gradient methods: Objective Function In Q-Learning we ...
Ray Walker's user avatar
14 votes
Accepted

What does the symbol $\mathbb E$ mean in these equations?

That's the Expected Value operator. Intuitively, it gives you the value that you would "expect" ("on average") the expression after it (often in square or other brackets) to have. Typically that ...
Dennis Soemers's user avatar
  • 10.1k
13 votes
Accepted

Why does DQN require two different networks?

My best guess that it's been done to reduce the computation time, otherwise we would have to find out the q value for each action and then select the best one. It has no real impact on computation ...
Neil Slater's user avatar
  • 30.2k
13 votes

Why doesn't Q-learning converge when using function approximation?

Here's an intuitive description answer: Function approximation can be done with any parameterizable function. Consider the problem of a $Q(s,a)$ space where $s$ is the positive reals, $a$ is $0$ or $...
John Doucette's user avatar
13 votes

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN). Tabular Q-learning (TQL) Deep Q-learning (DQL) ...
nbro's user avatar
  • 39.5k
12 votes

Can Q-learning be used for continuous (state or action) spaces?

Q-learning for continuous state spaces Yes, this is possible, provided you use some mechanism of approximation. One approach is to discretise the state space, and that doesn't have to reduce the ...
Neil Slater's user avatar
  • 30.2k
10 votes

How do we prove the n-step return error reduction property?

Let's start by looking at: $$\max_s \Bigl\lvert \mathbb{E}_{\pi} \left[ G_{t:t+n} \mid S_t = s \right] - v_{\pi}(s) \Bigr\rvert.$$ We can rewrite this by plugging in the definition of $G_{t:t+n}$: \...
Dennis Soemers's user avatar
  • 10.1k
10 votes
Accepted

What exactly is the advantage of double DQN over DQN?

In $Q$-learning there is what is known as a maximisation bias. That is because the update target is $r + \gamma \max_a Q(s,a)$. If you slightly overestimate your $Q$-value then this error gets ...
David's user avatar
  • 4,665
9 votes

Is Q-learning a type of model-based RL?

Tabular Q-Learning does not explicitly create a model of the transition function. It does not generate any output that you can afterwards use as a function to predict what the next state s' will be ...
Dennis Soemers's user avatar
  • 10.1k
8 votes
Accepted

How should I model all available actions of a chess game in deep Q-learning?

To model chess as a Markov decision problem (MDP) you can refer to the AlphaZero paper (Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm). The exact details can ...
Hai Nguyen's user avatar
8 votes

Are Q-learning and SARSA the same when action selection is greedy?

If we write the pseudo-code for the SARSA algorithm we first initialise our hyper-parameters etc. and then initialise $S_t$, which we use to choose $A_t$ from our policy $\pi(a|s)$. Then for each $t$ ...
David's user avatar
  • 4,665
8 votes
Accepted

Is Q-learning only capable of learning a deterministic policy?

If we assume a tabular setting, then Q-learning converges to the optimal state-action value function, from which an optimal policy can be derived, provided a few conditions are met. In finite MDPs, ...
nbro's user avatar
  • 39.5k
7 votes
Accepted

Is the discount not needed in a deterministic environment for Reinforcement Learning?

The motivation for adding the discount factor $\gamma$ is generally, at least initially, based simply in "theoretical convenience". Ideally, we'd like to define the "objective" of an RL agent as ...
Dennis Soemers's user avatar
  • 10.1k
7 votes
Accepted

My DQN is stuck and can't see where the problem is

After some research and reading this post, I see where my problem was: I was introducing a full consecutive batch of experiences, selected randomly, yes, but the experiences in the batch were ...
JCP's user avatar
  • 173
7 votes
Accepted

Can Q-learning be used in a POMDP?

The usual (as presented in Reinforcement Learning: An Introduction) $Q$-learning and SARSA algorithms use (and update) a function of a state $s$ and action $a$, $Q(s, a)$. These algorithms assume that ...
nbro's user avatar
  • 39.5k
7 votes

Why doesn't Q-learning converge when using function approximation?

As far as I'm aware, it is still somewhat of an open problem to get a really clear, formal understanding of exactly why / when we get a lack of convergence -- or, worse, sometimes a danger of ...
Dennis Soemers's user avatar
  • 10.1k
7 votes
Accepted

Does AlphaZero use Q-Learning?

Note: you mentioned in the comments that you are reading the old, pre-print version of the paper describing AlphaZero on arXiv. My answer will be for the "official", peer-reviewed, more recent ...
Dennis Soemers's user avatar
  • 10.1k
7 votes

What does the symbol $\mathbb E$ mean in these equations?

$\mathbb E$ is the symbol for the expectation (or expected value). To fully understand the concept of expected value, you need to understand the concept of random variable. An example should help ...
nbro's user avatar
  • 39.5k
7 votes
Accepted

What are some online courses for deep reinforcement learning?

Let me first say that deep RL is just the combination of RL with deep learning. So, if you study RL and deep learning, then studying deep RL should be straightforward. For this reason, this answer ...
nbro's user avatar
  • 39.5k
7 votes
Accepted

Is there any good reference for double deep Q-learning?

If you're interested in the theory behind Double Q-learning (not deep!), the reference paper would be Double Q-learning by Hado van Hasselt (2010). As for Double deep Q-learning (also called DDQN, ...
user5093249's user avatar
7 votes

Why do we need target network in deep Q learning?

In DQN that was presented in the original paper the update target for the Q-Network is $\left(r_t + \max_aQ(s_{t+1},a;\theta^-) - Q(s_t,a_t; \theta)\right)^2$ were $\theta^-$ is some old version of ...
David's user avatar
  • 4,665
7 votes
Accepted

What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, $Q(s, a)$. This is defined as $$Q(s, a) = \mathbb{E}_\pi\left[ G_t | S_t = s, ...
David's user avatar
  • 4,665
7 votes

Deep Q-Learning "catastrophic drop" reasons?

This is a case of overfitting the Q function leading to compounding errors when selecting actions. You have been training your policy for too long on the same data distribution. Overfitting Q ...
devidduma's user avatar
  • 532
6 votes
Accepted

Deep Q-Learning poor convergence on Stochastic Environment

The inputs that you describe seem like they should be sufficient for a DQN-based agent to learn a good strategy for playing Minesweeper, regardless of whether or not the starting layout changes. The ...
Dennis Soemers's user avatar
  • 10.1k
6 votes
Accepted

How do updates in SARSA and Q-learning differ in code?

Picking actions and making updates should be treated as separate things. For Q-learning you also need to explore by using some exploration strategy (e.g. $\epsilon$-greedy). Steps for Q-learning: ...
Brale's user avatar
  • 2,336
6 votes

What is the difference between the $\epsilon$-greedy and softmax policies?

The $\epsilon$-greedy policy is a policy that chooses the best action (i.e. the action associated with the highest value) with probability $1-\epsilon \in [0, 1]$ and a random action with probability $...
nbro's user avatar
  • 39.5k
6 votes
Accepted

Why do my rewards reduce after extensive training using D3QN?

It is not 100% clear, but this seems like an instance of catastrophic forgetting. This is something that often impacts reinforcement learning. I have answered a very similar question on Data Science ...
Neil Slater's user avatar
  • 30.2k
5 votes

How does Q-learning work in stochastic environments?

How does Q learning handle this? Is the Q function only used during the training process, where the future states are known? And is the Q function still used afterwards, if that is the case? The ...
Dennis Soemers's user avatar
  • 10.1k

Only top scored, non community-wiki answers of a minimum length are eligible