Q-learning uses a table to store all state-action pairs. Q-learning is a model-free RL algorithm, so how could there be the one called Deep Q-learning, as deep means using DNN; or maybe the state-action table (Q-table) is still there but the DNN is only for input reception (e.g. turning images into vectors)?

Deep Q-network seems to be only the DNN part of the Deep Q-learning program, and Q-network seems the short for Deep Q-network.

Q-learning, Deep Q-learning, and Deep Q-network, what are the differences? May be there a comparison table between these 3 terms?


In Q-learning (and in general value based reinforcement learning) we are typically interested in learning a Q-function, $Q(s, a)$. This is defined as $$Q(s, a) = \mathbb{E}_\pi\left[ G_t | S_t = s, A_t = a \right]\;.$$

For tabular Q-learning, where you have a finite state and action space - note that in practice even the spaces being finite might not be enough to not use DQN, if e.g. your state space contains a large number, say $10^{10000}$, of states, then it might not be manageable to maintain a separate Q-function for each state-action pair - you can maintain a table lookup that maintains your current estimate of the Q-value.

When you have an infinite state space (and/or action space) then it becomes impossible to use a table, and so you need to use function approximation to generalise across states. This is typically done using a deep neural network due to their expressive power. As a technical aside, the Q-networks don't usually take state and action as input, but take in a representation of the state (e.g. a $d$-dimensional vector, or an image) and output a real valued vector of size $|\mathcal{A}|$, where $\mathcal{A}$ is the action space.

Now, it seems in your question that you're confused as to why you use a model (the neural network) when Q-learning is, as you rightly say, model-free. The answer here is that when we talk about Reinforcement Learnings being model-free we are not talking about how their value-functions or policy are parameterised, we are actually talking about whether the algorithms use a model of the transition dynamics to help with their learning. That is, a model free algorithm doesn't use any knowledge about $p(s' | s, a)$ whereas model-based methods look to use this transition function - either because it is known exactly such as in Atari environments, or it must need to be approximated - to perform planning with the dynamics.

  • $\begingroup$ what is that q-network output vector? kinda softmax prob values of all actions? $\endgroup$ – datdinhquoc Jan 22 at 11:00
  • 1
    $\begingroup$ no, just real valued numbers. If you look at the expectation, it is not a probability nor does it have any bounds, so the output of the network reflects this. $\endgroup$ – David Ireland Jan 22 at 11:04

Here is a table that attempts to systematically show the differences between tabular Q-learning (TQL), deep Q-learning (DQL), and deep Q-network (DQN).

Tabular Q-learning (TQL) Deep Q-learning (DQL) Deep Q-network (DQN)
Is it an RL algorithm? Yes Yes No (unless you use DQN to refer to DQL, which is done often!)
Does it use neural networks? No. It uses a table. Yes No. DQN is the neural network.
Is it a model? No No Yes (but usually not in the RL sense)
Can it deal with continuous state spaces? No (unless you discretize them) Yes Yes (in the sense that it can get real-valued inputs for the states)
Can it deal with continuous action spaces? Yes (but maybe not a good idea) Yes (but maybe not a good idea) Yes (but only the sense that it can produce real-valued outputs for actions).
Does it converge? Yes Not necessarily Not necessarily
Is it an online learning algorithm? Yes No, if you use experience replay No, but it can be used in an online learning setting
  • 2
    $\begingroup$ Note that, when I say that neural networks are not online learning algorithms, I'm viewing neural networks as models and not algorithms. However, given that a specific neural network (with specific weights) computes some function and functions are algorithms (see lambda calculus), they could also be considered algorithms. In most cases, the best way to think of a neural network is as a model (i.e. something that you can use to approximate the desired function), but, sometimes, it's also useful to think of a neural network as a function. $\endgroup$ – nbro Jan 22 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.