3
$\begingroup$

I am wondering for which problem sizes a Deep Q-Learning algorithm is most appropriate. For example, whether it is particularly suited for low complexity problems or not for high complexity problems. And if that is the case, why?

$\endgroup$
1
  • $\begingroup$ Typically DQN works best with actions that have a continuous state space and a discrete, not too large action space. The complexity of the problem is more down to how expressive your model is. If you model is capable of learning a complex value manifold then DQN is likely able to obtain a good estimate of the values and thus provide a good policy. There are many updates to vanilla DQN to help learning too, such as a prioritised replay buffer. $\endgroup$
    – David
    Oct 12, 2022 at 10:37

1 Answer 1

0
$\begingroup$

Deep Q-learning is most suitable for problems with a large or infinite number of potential states. This is because the more states there are, the more information the Q-learning algorithm has to work with, and the more accurate its predictions about future states and rewards will be, with a large number of potential states the algorithm is able to learn from more data. In problems with fewer potential states, the algorithm may not have enough data to work with in order to make accurate predictions about future states and rewards.

Hence a large number of potential states is that the algorithm is able to generalize better. In problems with fewer potential states, the algorithm may not be able to learn the underlying structure of the problem as well, and thus may not be able to generalize as well to new states.

$\endgroup$
3
  • $\begingroup$ This is not quite right. If there are a small number of states then tabular Q-learning can be used in which it is guaranteed to converge to an optimal value function. Deep Q-learning is used, as you say, when state space is large or continuous, in which case it is impossible to store a tabular representation of the Q-values. $\endgroup$
    – David
    Oct 16, 2022 at 22:46
  • 1
    $\begingroup$ Yes, but in the Deep Q-learning case still the action is selected based on the max value of the Q-function, so either we get the optimal action always (under the assumption that the agent spends an infinity amount of time learning), or we do not know if the action chosen is the optimal action. But yes, in the tabular case it is guaranteed that the agent will eventually converge to the optimal action-value function. $\endgroup$
    – Faizy
    Oct 16, 2022 at 22:48
  • 1
    $\begingroup$ Sorry, I re-read again and see that you’re saying if the deep Q-learning had a small amount of data then it wouldn’t learn. I guess I would edit your answer to highlight you’re talking about deep Q-learning here (currently it is just Q-learning in bold). $\endgroup$
    – David
    Oct 16, 2022 at 22:59

You must log in to answer this question.