3
$\begingroup$

Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two reasons for this question:

  1. We can potentially obtain more samples from the learned model, which may speed up the learning speed.

  2. Models allow us to predict the future states and run simulations. This may lead to more valuable transitions, whereby speeding up learning.

But I heavily doubt this is the whole story. Sincerely, I hope someone could share a more detailed explanation for this question.

$\endgroup$
2
$\begingroup$

In this Medium article I found [1] it is quite well explained what is behind the better model efficiency in model based RL in comparison to model free one.

Main difference between those two is like you said, that the model helps in finding the correct path more efficiently because of the existence of the model. Maybe you cannot find new samples (point 1) but you know better the whole inner logic about the system and instead of just knowing what to do with the specific sample, you can relate it to the whole picture (sort of like in point 2: you can play with the choices) and make more profound calculations.

The article had a comparison, which told that you are writing a map in a city about every possible direction you can take when you are model-based and while in model-free you can enter specific places and remember which direction was best based on last visits but you still never know where you are coming nor going exactly.

In other words, if you think you're teaching a taxi driver on a big city with many signs and rules, model based guy would drive more precisely sooner because the sign language (the inner logic and model of the city) helps them on understanding the map sooner than just reacting crossing by crossing somewhat by chance all the time.

Sample efficiency tells what is the amount of information fetched from one sample [2]. Model-based machine can adjust model, maybe make some calculations about expected rewards AND after that the same as model-free, adjust the common policy. Model-free does only have the policy. Again the taxi guys: model-free guys know that last time and second last I stopped in crossing, model-based guy knows also it was due to red lights in pole. Third time model-free guy is the first in row and BANG - hits the crossing car. Next time rule is there comes sometimes cars and model-based guy knew that from the first place.


My sources:

[1] https://medium.com/the-official-integrate-ai-blog/understanding-reinforcement-learning-93d4e34e5698

[2] What is sample efficiency, and how can importance sampling be used to achieve it?

$\endgroup$

protected by nbro Aug 31 at 23:30

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.