Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions

Question

Sutton-Barto, page 164:

In the pseudocode algorithm for Dyna-Q in the box below, Model(s, a) denotes the contents of the model (predicted next state and reward) for state–action pair (s, a). Direct reinforcement learning, model learning, and planning are implemented by steps (d), (e), and (f), respectively. If (e) and (f) were omitted, the remaining algorithm would be one-step tabular Q-learning

If we look at Dyna-Q algorithm in the second figure, we see that "model learning" part (part e) is simply recording the real transitions and hence this is definitely not learning a model. If you do not learn a model, then how can we use the terms "model predictions" and "simulated experience" in the first figure? The simulated experience is the same as the real experience.

Answer 1

The figure is of a broad architecture "Dyna". Of which, Dyna-Q is one such variant. So I don't think it's required that all nomenclature be exact for Dyna-Q but I will proceed to defend it. On a previous page it is stated "The model-learning method is also table-based and assumes the environment is deterministic." If it is indeed deterministic, a single sample of the environment is a model of the environment for that state/action. "Simulated" in this case is a perfect simulation of a state the agent is not presently in.

Dyna-Q applied to a non-deterministic environment would of course model the environment poorly and depending on the amount of non-determinism would not be suitable.

Answer 2

The key property of a model is that it makes predictions of a system. Given some input - in Dyna a state and action - it provides an output, e.g. a predicted immediate reward and next state.

A random guess is technically a model, but normally not a useful one.

A reference list of previous results is a useful model when the same inputs can be reused. Dyna-Q deliberately targets the known parts of the model for "background planning".

As an aside, the process is almost identical to experience replay used in DQN method, where the "model" is renamed "replay buffer". This is done for partially different motivation than Dyna-Q, but it does show a strong conceptual link between planning and learning processes in reinforcement learning. In fact it's often not clear whether you should call some use of RL learning or planning - they are labels describing intent perhaps, and not necessarily strictly separate.