What is the difference between Q-learning, Deep Q-learning and Deep Q-network?

Question

Q-learning uses a table to store all state-action pairs. Q-learning is a model-free RL algorithm, so how could there be the one called Deep Q-learning, as deep means using DNN; or maybe the state-action table (Q-table) is still there but the DNN is only for input reception (e.g. turning images into vectors)?

Deep Q-network seems to be only the DNN part of the Deep Q-learning program, and Q-network seems the short for Deep Q-network.

Q-learning, Deep Q-learning, and Deep Q-network, what are the differences? May be there a comparison table between these 3 terms?

Answer 1

In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, $Q(s, a)$. This is defined as $$Q(s, a) = \mathbb{E}_\pi\left[ G_t | S_t = s, A_t = a \right]\;.$$

For tabular Q-learning, where you have a finite state and action space you can maintain a table lookup that maintains your current estimate of the Q-value. Note that in practice even the spaces being finite might not be enough to not use DQN, if e.g. your state space contains a large number, say $10^{10000}$, of states, then it might not be manageable to maintain a separate Q-function for each state-action pair

When you have an infinite state space (and/or action space) then it becomes impossible to use a table, and so you need to use function approximation to generalise across states. This is typically done using a deep neural network due to their expressive power. As a technical aside, the Q-networks don't usually take state and action as input, but take in a representation of the state (e.g. a $d$-dimensional vector, or an image) and output a real valued vector of size $|\mathcal{A}|$, where $\mathcal{A}$ is the action space.

Now, it seems in your question that you're confused as to why you use a model (the neural network) when Q-learning is, as you rightly say, model-free. The answer here is that when we talk about Reinforcement Learnings being model-free we are not talking about how their value-functions or policy are parameterised, we are actually talking about whether the algorithms use a model of the transition dynamics to help with their learning. That is, a model free algorithm doesn't use any knowledge about $p(s' | s, a)$ whereas model-based methods look to use this transition function - either because it is known exactly such as in Atari environments, or it must need to be approximated - to perform planning with the dynamics.

Answer 2

Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN).

	Tabular Q-learning (TQL)	Deep Q-learning (DQL)	Deep Q-network (DQN)
Is it an RL algorithm?	Yes	Yes	No (unless you use DQN to refer to DQL, which is done often!)
Does it use neural networks?	No. It uses a table.	Yes	No. DQN is the neural network.
Is it a model?	No	No	Yes (but usually not in the RL sense)
Can it deal with continuous state spaces?	No (unless you discretize them)	Yes	Yes (in the sense that it can get real-valued inputs for the states)
Can it deal with continuous action spaces?	Yes (but maybe not a good idea)	Yes (but maybe not a good idea)	Yes (but only the sense that it can produce real-valued outputs for actions).
Does it converge?	Yes	Not necessarily	Not necessarily
Is it an online learning algorithm?	Yes	No, if you use experience replay	No, but it can be used in an online learning setting