For example, in this article: https://www.learndatasci.com/tutorials/reinforcement-q-learning-scratch-python-openai-gym/, which explains Q-learnig, teaches the Smartcab problem, it has a visual environment of a 5x5 grid, where the agent will walk.

But I had this doubt: Is Q-learning limited to just visual scenarios, or is it much broader and can it be used to solve non-visual problems as well?


1 Answer 1


Reinforcement learning (RL) is a very broad description that applies to many variations of goal-driven learning problems. The key factor is being able to map the problem into a Markov Decision Process (MDP), at least in theory. You don't need to know all the parameters of an MDP model - just that there is a theoretical map from MDP to your problem. There is also no need to have a visual representation of a problem, although often that is a useful tool for humans looking at the same problem and assessing the performance of any agent.

One common way this requirement to match an MDP matters, is that the state variables need to represent data that is useful, and reasonably complete description of the environments. Specifically it needs to be enough data that it is theoretically possible to predict accurately what the distribution of next state and reward will become upon taking an action. This is called the Markov property.

Other than needing the Markov property (and even that is semi-optional for some problems), then state variables can be almost anything. They can represent physical position, velocity, money, cards in a pack, chemicals in a jar, the text of a novel.

Q-learning is a specific RL method for finding optimal behaviour in systems that are interacting according to MDP rules (at least in theory, even if you may need to discover what rules are in play). As such, it can cope with most types of state features.

Basic Q-learning with a table of action values does have limits. Not by type, but by amount. Very large state spaces become practically impossible to solve using tabular solutions. However, there are ways to fix this issue - sticking with Q-learning, you can extend it to Deep Q Networks (DQN), which can cope with very complex and large state spaces in principle. And those state spaces can be still be made out of almost any kind of variable.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .