5
$\begingroup$

Why do model-based methods use fewer samples than model-free methods? Here, I'm specifically referring to model-based methods in which we have to learn a policy and model. I can only think of two reasons for this question:

  1. We can potentially obtain more samples from the learned model, which may speed up the learning speed.

  2. Models allow us to predict the future states and run simulations. This may lead to more valuable transitions, whereby speeding up learning.

But I heavily doubt this is the whole story. Sincerely, I hope someone could share a more detailed explanation for this question.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

In this Medium article I found [1] it is quite well explained what is behind the better model efficiency in model based RL in comparison to model free one.

Main difference between those two is like you said, that the model helps in finding the correct path more efficiently because of the existence of the model. Maybe you cannot find new samples (point 1) but you know better the whole inner logic about the system and instead of just knowing what to do with the specific sample, you can relate it to the whole picture (sort of like in point 2: you can play with the choices) and make more profound calculations.

The article had a comparison, which told that you are writing a map in a city about every possible direction you can take when you are model-based and while in model-free you can enter specific places and remember which direction was best based on last visits but you still never know where you are coming nor going exactly.

In other words, if you think you're teaching a taxi driver on a big city with many signs and rules, model based guy would drive more precisely sooner because the sign language (the inner logic and model of the city) helps them on understanding the map sooner than just reacting crossing by crossing somewhat by chance all the time.

Sample efficiency tells what is the amount of information fetched from one sample [2]. Model-based machine can adjust model, maybe make some calculations about expected rewards AND after that the same as model-free, adjust the common policy. Model-free does only have the policy. Again the taxi guys: model-free guys know that last time and second last I stopped in crossing, model-based guy knows also it was due to red lights in pole. Third time model-free guy is the first in row and BANG - hits the crossing car. Next time rule is there comes sometimes cars and model-based guy knew that from the first place.

My sources:

[1] https://medium.com/the-official-integrate-ai-blog/understanding-reinforcement-learning-93d4e34e5698

[2] What is sample efficiency, and how can importance sampling be used to achieve it?

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .