Timeline for What's the difference between model-free and model-based reinforcement learning?
Current License: CC BY-SA 4.0
21 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 2, 2023 at 11:43 | comment | added | Neil Slater | @Sam - the learning system in that case must be model-based, yes. Without a model, TD learning using state values cannot make decisions. You cannot run value-based TD learning in a control scenario otehrwise, which is why you would typically use SARSA or Q learning (which are TD learning on action values) if you want a model-free TD learner. TD on state values still works model-free in predicion scenarios though. | |
| Feb 2, 2023 at 9:53 | comment | added | Sam | @NeilSlater so is the statement 'Basic TD learning, using state values only, must also be model-based in order to work as a control system and pick actions.' still correct (I was confused by it)? | |
| Feb 2, 2023 at 9:37 | comment | added | Neil Slater | @Sam: Yes TD learning is itself model-free. It doesn't require any model. However, it is often used in situations where there is a model in the combined system. E.g. DynaQ. | |
| Feb 2, 2023 at 3:39 | comment | added | Sam | Other sources suggest TD-learning is model-free though. | |
| S Jun 11, 2022 at 11:26 | history | suggested | Jonathan Rayner | CC BY-SA 4.0 |
Fixed a broken link to "imagination-based agents"
|
| Jun 11, 2022 at 10:58 | review | Suggested edits | |||
| S Jun 11, 2022 at 11:26 | |||||
| Jul 31, 2021 at 13:11 | comment | added | Neil Slater | @HermesMorales There is a complex relationship between models, simulation and planning, in terms of when you might consider that you are using one or the other. One important detail is whether you have a sampling model or a distribution model. Model-free methods are often paired with simulations which are effectively sampling models. If the end goal is to then use the trained agent in a real environment, you might consider the training session as planning | |
| Jul 31, 2021 at 11:26 | comment | added | Hermes Morales | Continuation: You also need to construct a correct "real world simulation", except if you can experiment in the real world. In this sense, RL is not model free. Am I wright? | |
| Jul 31, 2021 at 11:26 | comment | added | Hermes Morales | @NeilSlater You say that "if you have a real-world problem in an environment without an explicit known model at the start, then the safest bet is to use a model-free approach such as ". You mean that you should use real world simulations? usually, when using a simulation even if the "learning agent" does not use the model, the model is necessary -and present-in order to excute the script and study the algorithm performance. So, apparently it is correct to say that in real world problem not only the "learning algorithm" is the problem. | |
| Apr 29, 2021 at 15:03 | comment | added | Neil Slater | @haneulkim It is the prediction of next reward and state that are part of a model of the environment. The action is treated separately, using a policy of some kind, and yes all learning agents must have a method to do that somehow. The difference is important: An agent can control its action choice, but cannot control any other aspect of the environment rules. When looking at the difference between action taken (exploring behaviour policy) and the value update (greedy target policy), it doesn't matter whether you consider this "generated", it has nothing to do with the model of the environment | |
| Apr 29, 2021 at 9:07 | comment | added | haneulkim | Thank you for informative answer! I want to clarify one thing, for model-free algorithms you've said they "never use generated predictions of next state and next reward to alter behaviour" however in Q-learning agent may take greedy action(isn't this generated?) or random action even though it updates Q-value assuming agent took greedy action, isn't greedy action generated since it is not actually taken by the agent? | |
| May 31, 2020 at 8:40 | comment | added | Neil Slater | @d56: I think that could be a new question on the site. In brief, you can still use most RL methods, and add planning to refine estimates of return values. | |
| May 31, 2020 at 0:37 | comment | added | d56 | @NeilSlater, what would you recommend for an environment which is fully observable, but also gives back a model of it -- the exact reward and transition functions are known. I think there could be a benefit in using something beyond DQN or A3C to take advantage of that knowledge. I cannot use dynamic programming since state-space is too large. thanks | |
| Dec 1, 2019 at 20:58 | comment | added | Miguel Saraiva | I could be wrong, I'm a novice in the area. I just remember a teacher from the field making that comment after I had done the same remark. | |
| Dec 1, 2019 at 20:50 | comment | added | Neil Slater | @MiguelSaraiva: I'm not 100% certain about that, but have removed the reference to MCTS. Out of interest, where would you place DynaQ regarding this limitation of the usage of the terms? I think it becomes tricky, when the algorithms all share such a common view of the MDP model and improving policies, to tell where the boundaris are between planning and learning. | |
| Dec 1, 2019 at 20:47 | history | edited | Neil Slater | CC BY-SA 4.0 |
deleted 36 characters in body
|
| Dec 1, 2019 at 20:44 | comment | added | Miguel Saraiva | A small correction, normally the terms "model based" or "model free" are not used for planning algorithms such as MCTS. It is only used to classify learning algorithms. | |
| Nov 7, 2018 at 9:36 | history | edited | Neil Slater | CC BY-SA 4.0 |
added 36 characters in body
|
| Nov 7, 2018 at 9:23 | history | edited | Neil Slater | CC BY-SA 4.0 |
added 1913 characters in body
|
| Jun 13, 2018 at 13:08 | history | edited | Neil Slater | CC BY-SA 4.0 |
added 625 characters in body
|
| Jun 13, 2018 at 12:47 | history | answered | Neil Slater | CC BY-SA 4.0 |