0
$\begingroup$

Sutton-Barto, page 163:

enter image description here

Figure 8.1: The general Dyna Architecture. Real experience, passing back and forth between the environment and the policy, affects policy and value functions in much the same way as does simulated experience generated by the model of the environment.

The central arrow is unnecessary in my opinion and moreover, the explanation for it is not clear to me. Could someone clarify it? Why real experience passing back and forth between the environment and the policy? The direction from "real experience" to "environment" is not clear to me. Moreover, the direction from "real experience" to "Polcy/value functions" is already covered in direct RL update.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The diagram should not be read as a formal rendering of process or architecture. It's not a UML description or similar. It is a visual aid to the text description.

The large arrow shows dependency of data collection that's core to the whole process.

The environment responds to actions chosen by the policy.

The policy is informed only by data collected from the environment.

The smaller arrows show data flow between sub-components, and how Dyna-Q is constructed to learn and improve policy. From a 10,000 foot view though, the large arrow reminds you that the usual interdependence of agent and environment is in play, and you still have usual impact of that - exploration Vs exploitation tradeoff for example.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .