I ask because if RL is indeed an MDP, then there should be absolutely no problem with training an agent on any available episode roll-out data, right? Because an MDP implies for any state S, the optimal action to take is entirely dependent on the state.

So while I can understand the argument from prioritized experience replay that some data is more valuable for learning than others. I cannot understand how old data would be harmful.

Further Clarification:

The emphasis on an MDP is perhaps misleading. I don't mean to imply that optimal control should be dependent on previous states necessarily... Similarly, the emphasis on off-policy & on-policy is perhaps better placed on online vs offline RL.

Performance Gap:

This [offline RL] setting is harder as it removes the ability for agents to explore the environment and collect additional feedback.

Consider a simple example: with Q-table 'learning'. It seems to me that Online-RL is analogous to deleting perfectly valid entries from the Q-table in order to reap better performance. This seems to me like a paradox and that's what confuses me.

  • $\begingroup$ I am not sure I understand this question and, in particular, I don't understand this "I ask because if optimal control were indeed a MDP" and "performance gap between off/on-policy RL". What "gap" between on and off-policy RL are you talking about? A gap in terms of what? MDP is just a way to formulate a problem. Optimal control is an area of study. Also, "I cannot understand how old data would be harmful." doesn't seem to be related to the title. Please, clarify what you're really asking here. $\endgroup$
    – nbro
    Jan 6 at 10:02
  • $\begingroup$ @nbro Neil Slater understood what I was asking so you can see his answer for context. I'm saying that off policy/offline does worse than on-policy/online, and I'm saying that I don't get why because formulating the problem as a MDP implies that any episode rollout data will always remain valid training data. So in theory excluding old data like online-RL does shouldn't increase performance yet it seems it does. $\endgroup$
    – profPlum
    Jan 6 at 15:07
  • $\begingroup$ @profPlum: I need to understand what you mean by "performance" and what examples you have that online is "better" than offline by that measure? It surely is not by all measures - do have examples, or links to relevant statistics? Also, if my answer has not resolved your question, it would be useful to get some feedback on what you are still stuck on $\endgroup$ Jan 6 at 15:53
  • $\begingroup$ @NeilSlater I didn't have a specific metric in mind, but lets say expected accumulated reward per episode after training. Please see the citation/quote I have from Pieter Abeel stating that offline RL is 'harder'. Also you have a pretty good answer, I may accept it but just wanted to wait & see what other people say first. $\endgroup$
    – profPlum
    Jan 6 at 15:58
  • $\begingroup$ @nbro I edited the original question to try to state what I mean more clearly. I'm sorry I probably shouldn't have placed emphasis on optimal control & MDPs. I think the Q-table example I gave is perhaps a better way of stating the question. $\endgroup$
    – profPlum
    Jan 6 at 16:00

there should be absolutely no problem with training an agent on any available episode roll-out data. That is because a MDP implies for an any state S, the optimal action to take is entirely dependent on the state.

The desired end-state of the trained model is that it can identify the optimal action. When comparing reinforcement learning (RL) methods, you are looking at how they use data from experience. This data does not directly identify optimal actions, and all RL control approaches perform significant processing of the raw data in order to learn about the environment and make this identification.

Any rollout may contain useful data that can be learned from for control, but it requires processing in order to drive a meaningful policy. When comparing on-policy and off-policy, you are looking at two different approaches to process this data.

The differences between on-policy and off-policy methods are to do with the effectiveness of the learning algorithm in particular scenarios, and not related to using MDPs as a framework. Instead the existence of the two approaches (and other ways to solve reinforcement learning problems) are a logical consequence of the MDP framework.

I cannot understand how old data would be harmful.

When learning for optimal control, then the policy must change in order to improve. Data collected under old policies has a different distribution and results from data expected under the current policy. You can often adjust for this and still learn something (which may or may not be useful to the rest of the task) - that is exactly what off-policy learning does.

Any data can be harmful in machine learning if your learning model makes incorrect assumptions about it. That is the case when an on-policy learning algorithm attempts to use data collected under a different policy. This also applies to many other types of machine learning - if something is different in an old data set to your target distribution, the trained model will be biased towards old data and make less accurate predictions on new examples.

if optimal control were indeed a MDP, then there should be absolutely no problem with training an agent on any available episode roll-out data

This is not true in general. Even off-policy algorithms need there to be some useful relationship to the current target policy. Data from completely random rollouts in complex problems is rarely useful for learning, except as an initial bootstrap when nothing is known about the environment.

Assuming there is some useful data that could be extracted, then there is no problem training an agent, provided that agent adjusts correctly for the difference in policy. That requires off-policy agents, because on-policy agents do not have the ability to make the adjustment.


Not the answer you're looking for? Browse other questions tagged or ask your own question.