Why do we need a model of the environment in Dyna?

Question

In chapter 8 of "Reinforcement Learning: An Introduction" by Sutton and Barto, it is stated that Dyna needs a model to simulate the environment.

But why do we need a model? Why can't we just use the real environment itself? Wouldn't it be more helpful to use real environment instead of fake one?

Answer 1

Unlike algorithms presented in other chapters of Sutton and Barto, Dyna is a planning algorithm. That means that it makes decisions online, in a real environment, that attempt to be as optimal as possible given some constraints such as current knowledge and time available to compute between time steps. This differs from learning-only online algorithms which typically take a small step towards optimality on each piece of new experience as it happens.

A planning algorithm can only do its job well if it is allowed to "look ahead" at the consequences of its behaviour whilst learning online. In fact, this is the definition of planning - to choose an action based on reasoning about consequences of that action.

For an algorithm to look ahead before taking an action, it needs a model of how the environment will respond to that action. That model does not need to be coded up directly - e.g. you don't necessarily need to write a physics engine to predict the real world (although a basic one might be a good prior or pre-training step). Instead it can be a learned model, and typically in e.g. Dyna-Q, that is what you use.

There is a strong relation between Dyna-Q, and regular Q-learning with experience replay. In the most basic forms, they are essentially the same algorithm with a different framing. However, you can take the planning ideas further e.g. focus improvements around the currently experienced state and paths to a goal state in Dyna-Q, perhaps making it closer to MCTS conceptually.

Wouldn't it be more helpful to use real env instead of fake one?

Most real environments do not let you take actions, see the consequences and then rewind in order to re-try. Essentially that is what planning algorithms are making up for - they try to predict consequences. This is important when mistakes made during training have real consequences, for example for a physical robot navigating an environment where there might be a possibility of a fall or collision that damaged something. Whilst online learning algorithms such as SARSA will also help with this in different ways (in SARSA by changing policy to allow for exploratory moves), typically Q-learning will be weaker than Dyna-Q when it comes to learning quickly from mistakes. With the usual caveat: Much still depends on the specific problem and choices of hyperparameters.

Answer 2

Sutton's Dyna has been shown to be more effective for many problem spaces than learning systems that work without a model, yet it requires fewer processing cycles than certainty-equivalence methods. It is advanced in that it, in parallel, builds a model and adjusts behavioral policy based on both incoming information and the model. The goal was to integrate both identified capabilities of the human brain.

Why do we need Model in Dyna?

The capacity to model is an essential part of the Dyna architecture and may, over time, prove to be an essential component to achieve the greater effectiveness. Many think so, myself included. In other words, there may be no equivalently effective mechanism than building and maintaining a model for many problem sets.

Why can't we just utilize a real environment itself? Wouldn't it be more helpful to use real environment instead of fake one?

The real environment cannot be placed in memory for many reasons. Primarily, it would not fit. Furthermore, not much of it can be acquired. Only images of the environment can be acquired and placed in memory. The most important characteristic of images, whether they be stock prices, temperature readings, or streamed video, is that they are grossly sparse and undifferentiated representations of the environment on which we are attempting to operate.