2
$\begingroup$

I have read most of Sutton and Barto's introductory text on reinforcement learning. I thought I would try to apply some of the RL algorithms in the book to a previous assignment I had done on Sokoban, in which you are in a maze-like grid environment, trying to stack three snowballs into a snowman on a predefined location on the grid.

The basic algorithms (MC control, Q-learning, or Dyna-Q) seemed to all be based on solving whichever specific maze the agent was trained on. For example, the transition probabilities of going from coordinate (1,2) to (1,3) would be different for different mazes (since in one maze, we could have an obstacle at (1,3)). An agent that calculates its rewards based on one maze using these algorithms doesn't seem like it would know what to do given a totally different maze. It would have to retrain: 1) either take real life actions to relearn from scratch how to navigate a maze, or 2) be given the model of the maze, either exact or approximate (which seems infeasible in a real life setting) so that planning without taking actions is possible.

When I started learning RL, I thought that it would be more generalizable. This leads me to the question: Is this problem covered in multi-task RL? How would you categorize the various areas of RL in terms of the general problem that it is looking to solve?

$\endgroup$
2
$\begingroup$

The basic algorithms (MC control, Q-learning, or Dyna-Q) seemed to all be based on solving whichever specific maze the agent was trained on.

All RL algorithms are based on creating solutions to a defined state and action space. If you limit your state space representation and training to a single maze, then that is what will be learned. This is no different from other machine learning approaches - they learn the traits of a population by being shown samples from that population (not just one example). They also need to be built for the range of input parameters that you need them to solve.

In the case of RL, and your maze solver, that means the state representation needs to cover all possible mazes, not just a location in a single maze (there are ways to internalise some of the representation to the learning process such as using RNNs, but that is not relevant to the main answer here).

The toy environments in Sutton & Barto are often trivial to solve using non-RL approaches. They are not demonstrations of what RL can do, instead they have been chosen to explain how a particular issue related to learning works. Sutton & Barto does include a chapter on more interesting and advanced uses of RL - that is chapter 16 "Applications and Case Studies" in the second edition.

When I started learning RL, I thought that it would be more generalizable.

It is, but without some kind of pre-training to support generalisation from a low number of examples, you have to:

  • Model the general problem

  • Train the agent on the general problem

RL agents trained from scratch on new problems can seem very inefficient compared to learning by living creatures that RL roughly parallels. However, RL is not a model for general intelligence, but for learning through trial and error. Most examples start from no knowledge, not even basic priors for a maze such as the grid layout or the generalisable knowledge of movement and location.

If you do provide a more general problem definition and training examples, and use a function approximator that can generalise internally (such as a neural network), then an agent can learn to solve problems in a more general sense and may also generate internal representations that (approximately) match up to common factors in the general problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.