Can I treat "experience" in reinforcement learning as "training data" in statistical learning?

Question

Statistics is a branch of mathematics that extracts useful information from data. The data is generally called as "training data" in statistical (machine) learning.

Consider the following paragraph from the section 1.1 Reinforcement Learning of CHAPTER 1. THE REINFORCEMENT LEARNING PROBLEM of the textbook Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto.

Reinforcement learning is different from supervised learning, the kind of learning studied in most current research in field of machine learning. Supervised learning is learning from a training set of labeled examples provided by a knowledgable external supervisor. Each example is a description of a situation together with a specification—the label—of the correct action the system should take to that situation, which is often to identify a category to which the situation belongs. The object of this kind of learning is for the system to extrapolate, or generalize, its responses so that it acts correctly in situations not present in the training set. This is an important kind of learning, but alone it is not adequate for learning from interaction. In interactive problems it is often impractical to obtain examples of desired behavior that are both correct and representative of all the situations in which the agent has to act. In uncharted territory—where one would expect learning to be most beneficial—an agent must be able to learn from its own experience.

You can observe that, training data in machine learning, if we model it in a proper format, can be for reinforcement learning. But, may not be complete and practical.

I am asking this question in the view of statistical learning rather than machine learning alone. training data in statistical learning can be understood to any data, at any time instant, useful in learning.

Then, is it perfectly fine to always interpret experience in reinforcement learning as training data in statistical learning?

Answer 1

The main similarity between reinforcement learning experience and supervised learning datasets, is that both consist of a set of records. These records are commonly expressed as vectors of numbers for use in the algorithms. In addition, reinforcement learning that uses neural networks (or other function approximation) will typically implement some variant of supervised learning internally.

There are a few key differences between a prepared dataset for supervised learning and the experience in reinforcement learning. There are exceptions to these, but these are the usual case:

• A supervised learning dataset has a fixed target value to learn by association, for each entry, e.g. a class or regression value. An individual reinforcement learning experience does not - the raw tuple of state, action, reward, next state $(s,a,r,s')$ must be processed in some way to obtain a useful training target value, and this processing is not static. When training in reinforcement learning for optimal control, even with historical experience, the target values must constantly be re-assessed.

• Reinforcement learning is in part a design for actively collecting experience. There is no equivalent in supervised learning where the dataset is a given.

• Reinforcement learning experience arrives in groups of $(s, a, r, s')$ - state, action, reward, next state - such groups of related data within a record are called tuples. The RL records are often correlated with each other, at least initially because each time step changes things only slightly, and that can be bad when combined with supervised learning which usually assumes uncorrelated data. In supervised learning you will often have a deliberate shuffling or randomisation of dataset order to protect against this. Experience replay in deep RL is a related idea to protect internal neural networks from being exposed to training samples in sequences with correlated values.

It is possible to apply reinforcement learning to supervised learning problems in theory. You can do this by making the agent guess each labelled value as an action, and reward it with negative the loss from the supervised learning. This is generally a bad idea because it is very inefficient, and there is no matching concept of state transitions in the supervised learning problem (the agent cannot impact what state is next due to its guess). However, the fact that this is possible with very little modification to the reinforcement learning agent shows how general reinforcement learning is as a learning approach.

The inverse is not really true, you cannot normally adjust a supervised learning algorithm so that it can solve a reinforcement learning problem from the given experience. There are some edge cases, such as when learning only from previous experience to assess an existing policy, or to learn a control algorithm which only cares about immediate reward. In which case you could use reinforcement learning theory to help construct a fixed dataset and give that to a supervised learning algorithm. However, by far the most common approach is to use supervised learning approaches (e.g. a neural network) as components of the agent, and rely on reinforcement learning to generate data for them on the fly.