How is trajectory sampling different than normal (importance) sampling in reinforcement learning?

Question

I am using Sutton and Barto's book for Reinforcement Learning.

In Chapter 8, I am having difficulty in understanding the Trajectory Sampling.

I have read the particular section on trajectory sampling (Sec 8.6) two times (plus 3rd time partially) but still, I do not get how it is different from the normal sampling update, and what are its benefits.

Answer 1

Here is my understanding:

In trajectory sampling as the book describes it, we use the current policy on the simulator to get (next-state, action) pairs. The advantage is that if some states occur more frequently than others in that environment, and if we take enough samples, the distribution among the samples would be similar to the actual distribution.

On the another hand, you can sample in a different manner if you have access to the model. Suppose you have access to the transition distribution. Then you can sample your start state uniformly, and use the transition distribution to get the (next-state, action) pairs. This can be useful if you want to force your algorithm to look at all states evenly, even if that's not the actual case.