# Can I treat "experience" in reinforcement learning as "training data" in statistical learning?

Statistics is a branch of mathematics that extracts useful information from data. The data is generally called as "training data" in statistical (machine) learning.

Consider the following paragraph from the section 1.1 Reinforcement Learning of CHAPTER 1. THE REINFORCEMENT LEARNING PROBLEM of the textbook Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto.

Reinforcement learning is different from supervised learning, the kind of learning studied in most current research in field of machine learning. Supervised learning is learning from a training set of labeled examples provided by a knowledgable external supervisor. Each example is a description of a situation together with a specification—the label—of the correct action the system should take to that situation, which is often to identify a category to which the situation belongs. The object of this kind of learning is for the system to extrapolate, or generalize, its responses so that it acts correctly in situations not present in the training set. This is an important kind of learning, but alone it is not adequate for learning from interaction. In interactive problems it is often impractical to obtain examples of desired behavior that are both correct and representative of all the situations in which the agent has to act. In uncharted territory—where one would expect learning to be most beneficial—an agent must be able to learn from its own experience.

You can observe that, training data in machine learning, if we model it in a proper format, can be for reinforcement learning. But, may not be complete and practical.

I am asking this question in the view of statistical learning rather than machine learning alone. training data in statistical learning can be understood to any data, at any time instant, useful in learning.

Then, is it perfectly fine to always interpret experience in reinforcement learning as training data in statistical learning?

• When you say "I am asking this question in the view of statistical learning rather than machine learning alone.", you need to clarify what you mean by both, because, from my perspective, statistical learning is just another name for ML, although you may put emphasis on statistics, or, put in another way, to me it's just ML with a statistical perspective, but given that ML and statistics overlap a lot, then the difference is really minimal if none and this is just a problem of consensus and/or terminology.
– nbro
Sep 5 '21 at 11:30
• Having said that, I am not sure I understand the type of answer you're looking for. My answer would be "Yes, experience would be some type of data. We call it "experience" because we can collect more and more, as opposed to SL, where you typically have a fixed dataset, while in RL, at least in simulation, this is not the case, although with things like experience replay, one may think of it as collecting a dataset, but this dataset is not fixed and can actually change anyway." I will not post this as a formal answer until you clarify the type of answer you're looking for.
– nbro
Sep 5 '21 at 11:33
• @nbro Although they are same, I am guessing that there may be some differences if we go too deep in to formal aspects. So. to keep them intact at all levels. I used two different names. Sep 5 '21 at 12:01
• If I always start interpreting experience in RL as the data in statistical AI, then will I get contradiction somewhere or not is the question. Sep 5 '21 at 12:02
• I do not understand this question, I think because you do not clarify anywahere what you mean by "treat like" from the title. Or "interpret" in the question body. Assuming there are some useful parallels between the two terms (and I think there are), what do you intend to do with that information? Are you looking for ways to transform between supervised learning and RL, or trying to understand the differences between RL experience and a supervised learning dataset? Sep 5 '21 at 13:06

The main similarity between reinforcement learning experience and supervised learning datasets, is that both consist of a set of records. These records are commonly expressed as vectors of numbers for use in the algorithms. In addition, reinforcement learning that uses neural networks (or other function approximation) will typically implement some variant of supervised learning internally.

There are a few key differences between a prepared dataset for supervised learning and the experience in reinforcement learning. There are exceptions to these, but these are the usual case:

• A supervised learning dataset has a fixed target value to learn by association, for each entry, e.g. a class or regression value. An individual reinforcement learning experience does not - the raw tuple of state, action, reward, next state $$(s,a,r,s')$$ must be processed in some way to obtain a useful training target value, and this processing is not static. When training in reinforcement learning for optimal control, even with historical experience, the target values must constantly be re-assessed.

• Reinforcement learning is in part a design for actively collecting experience. There is no equivalent in supervised learning where the dataset is a given.

• Reinforcement learning experience arrives in groups of $$(s, a, r, s')$$ - state, action, reward, next state - such groups of related data within a record are called tuples. The RL records are often correlated with each other, at least initially because each time step changes things only slightly, and that can be bad when combined with supervised learning which usually assumes uncorrelated data. In supervised learning you will often have a deliberate shuffling or randomisation of dataset order to protect against this. Experience replay in deep RL is a related idea to protect internal neural networks from being exposed to training samples in sequences with correlated values.

It is possible to apply reinforcement learning to supervised learning problems in theory. You can do this by making the agent guess each labelled value as an action, and reward it with negative the loss from the supervised learning. This is generally a bad idea because it is very inefficient, and there is no matching concept of state transitions in the supervised learning problem (the agent cannot impact what state is next due to its guess). However, the fact that this is possible with very little modification to the reinforcement learning agent shows how general reinforcement learning is as a learning approach.

The inverse is not really true, you cannot normally adjust a supervised learning algorithm so that it can solve a reinforcement learning problem from the given experience. There are some edge cases, such as when learning only from previous experience to assess an existing policy, or to learn a control algorithm which only cares about immediate reward. In which case you could use reinforcement learning theory to help construct a fixed dataset and give that to a supervised learning algorithm. However, by far the most common approach is to use supervised learning approaches (e.g. a neural network) as components of the agent, and rely on reinforcement learning to generate data for them on the fly.