In reinforcement learning (RL), what is the difference between training and testing an algorithm/agent? If I understood correctly, testing is also referred to as evaluation.
As I see it, both imply the same procedure: select an action, apply to the environment, get a reward, and next state, and so on. But I've seen that, e.g., the Tensorforce RL framework allows running with or without evaluation.
In reinforcement learning (RL), you typically imagine that there's an agent that interacts, in time steps, with an environment by taking actions. On each time step $t$, the agent takes the action $a_t \in \mathcal{A}$ in the state $s_t \in \mathcal{S}$, receives a reward (or reinforcement) signal $r_t \in \mathbb{R}$ from the environment and the agent and the environment move to another state $s_{t+1} \in \mathcal{S}$, where $\mathcal{A}$ is the action space and $\mathcal{S}$ is the state space of the environment, which is typically assumed to be a Markov decision process (MDP).
The goal is to find a policy that maximizes the expected return (i.e. a sum of rewards starting from the current time step). The policy that maximizes the expected return is called the optimal policy.
A policy is a function that maps states to actions. Intuitively, the policy is the strategy that implements the behavior of the RL agent while interacting with the environment.
A policy can be deterministic or stochastic. A deterministic policy can be denoted as $\pi : \mathcal{S} \rightarrow \mathcal{A}$. So, a deterministic policy maps a state $s$ to an action $a$ with probability $1$. A stochastic policy maps states to a probability distribution over actions. A stochastic policy can thus be denoted as $\pi(a \mid s)$ to indicate that it is a conditional probability distribution of an action $a$ given that the agent is in the state $s$.
The expected return can be formally written as
$$\mathbb{E}\left[ G_t \right] = \mathbb{E}\left[ \sum_{i=t+1}^\infty R_i \right]$$
where $t$ is the current time step (so we don't care about the past), $R_i$ is a random variable that represents the probable reward at time step $i$, and $G_t = \sum_{i=t+1}^\infty R_i $ is the so-called return (i.e. a sum of future rewards, in this case, starting from time step $t$), which is also a random variable.
In this context, the most important job of the human programmer is to define a function $\mathcal{R}: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$, the reward function, which provides the reinforcement (or reward) signal to the RL agent while interacting with the environment. $\mathcal{R}$ will deterministically or stochastically determine the reward that the agent receives every time it takes action $a$ in the state $s$. The reward function $R$ is also part of the environment (i.e. the MDP).
Note that $\mathcal{R}$, the reward function, is different from $R_i$, which is a random variable that represents the reward at time step $i$. However, clearly, the two are very related. In fact, the reward function will determine the actual realizations of the random variables $R_i$ and thus of the return $G_i$.
To estimate the optimal policy, you typically design optimization algorithms.
The most famous RL algorithm is probably Q-learning, which is also a numerical and iterative algorithm. Q-learning implements the interaction between an RL agent and the environment (described above). More concretely, it attempts to estimate a function that is closely related to the policy and from which the policy can be derived. This function is called the value function, and, in the case of Q-learning, it's a function of the form $Q : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$. The name $Q$-learning derives from this function, which is often denoted as $Q$.
Q-learning doesn't necessarily find the optimal policy, but there are cases where it is guaranteed to find the optimal policy (but I won't dive into the details).
Of course, I cannot describe all the details of Q-learning in this answer. Just keep in mind that, to estimate a policy, in RL, you will typically use a numerical and iterative optimization algorithm (e.g. Q-learning).
In RL, training (also known as learning) generally refers to the use of RL algorithms, such as Q-learning, to estimate the optimal policy (or a value function)
Of course, as in any other machine learning problem (such as supervised learning), there are many practical considerations related to the implementation of these RL algorithms, such as
These and other details (which, of course, I cannot list exhaustively) can actually affect the policy that you obtain. However, the basic goal during the learning or training phase in RL is to find a policy (possibly, optimal, but this is almost never the case).
During learning (or training), you may not be able to find the optimal policy, so how can you be sure that the learned policy to solve the actual real-world problem is good enough? This question needs to be answered, ideally before deploying your RL algorithm.
The evaluation phase of an RL algorithm is the assessment of the quality of the learned policy and how much reward the agent obtains if it follows that policy. So, a typical metric that can be used to assess the quality of the policy is to plot the sum of all rewards received so far (i.e. cumulative reward or return) as a function of the number of steps. One RL algorithm dominates another if its plot is consistently above the other. You should note that the evaluation phase can actually occur during the training phase too. Moreover, you could also assess the generalization of your learned policy by evaluating it (as just described) in different (but similar) environments to the training environment [1].
The section 12.6 Evaluating Reinforcement Learning Algorithms of the book Artificial Intelligence: Foundations of Computational Agents (2017) by Poole and Mackworth provides more details about the evaluation phase in reinforcement learning, so you should probably read it.
Apart from evaluating the learned policy, you can also evaluate your RL algorithm, in terms of
During training, you want to find the policy. During the evaluation, you want to assess the quality of the learned policy (or RL algorithm). You can perform the evaluation even during training.
The general workflow for using and applying reinforcement learning to solve a task is the following.
Training in Reinforcement learning employs a system of rewards and penalties to compel the computer to solve a problem by itself.
Human involvement is limited to changing the environment and tweaking the system of rewards and penalties.
As the computer maximizes the reward, it is prone to seeking unexpected ways of doing it.
Human involvement is focused on preventing it from exploiting the system and motivating the machine to perform the task in the way expected.
Reinforcement learning is useful when there is no “proper way” to perform a task, yet there are rules the model has to follow to perform its duties correctly.
Example: By tweaking and seeking the optimal policy for deep reinforcement learning, we built an agent that in just 20 minutes reached a superhuman level in playing Atari games.
Similar algorithms, in principle, can be used to build AI for an autonomous car.
Debugging RL algorithms is very hard. Everything runs and you are not sure where the problem is.
To test if it worked well, if the trained agent is good at what it was trained for, you take your trained model and apply it to the situation it is trained for.
If it’s something like chess or Go, you could benchmark it against other engines (say stockfish for chess) or human players.
You can also define metrics for performance, ways of measuring the quality of the agent’s decisions.
In some settings (e.g a Reinforcement Learning Pacman player), the game score literally defines the target outcome, so you can just evaluate your model’s performance based on that metric.
The goal of the reinforcement learning (RL) is to use data obtained via interaction with the environment to solve the underlying Markov Decision Process (MDP). "Solving the MDP" is tantamount to finding the optimal policy (with respect to the MDP's underlying dynamics which are usually assumed to be stationary).
Training is the process of using data in order to find the optimal policy. Testing is the process of evaluating the (final) policy obtained by training.
Note that, since we're generally testing the policy on the same MDP we used for training, the distinction between the training dataset and the testing set is no longer as important as it is the case with say supervised learning. Consequently, classical notions of overfitting and generalization should be approached from a different angle as well.
If you want, you can do training and testing in RL. Exactly the same usage, training for building up a policy, and testing for evaluation.
In supervised learning, if you use test data in training, it is like cheating. You cannot trust the evaluation. That's why we separate train and test data.
The Objective of RL is a little different. RL trying to find the optimal policy. Since RL collects the information by doing, while the agent explores the environment (for more information), there might be a loss in the objective function. But, it might be inevitable for a better future gain.
Multi-arm bandit example, If there are 10 slot machines. They will return random amounts of money. They have different expected returns. I want to find the best way to maximize my gain. easy, I have to find the machine with the greatest expected return and use only the machine. How to find the best machine?
If we have a training and testing (periods), For example, I will give you an hour of the training period, so it doesn't matter if you lose or how much you earn. And in the testing period, I will evaluate your performance.
What would you do? In the training period, you will try as much as possible, without considering the performance/gain. And in the testing period, you will use only the best machine you found.
This is not a typical RL situation. RL is trying to find the best way, Learning by doing. All the results while doing are considered.
Suppose that I tried all 10 machines once each. And, the No.3 machine gave me the most money. But I am not sure that it is the best machine, because all the machines provide a RANDOM amount. If I keep using the No.3 machine, it might be a good idea, because according to the information so far, it is the best machine. However, You might miss the better machine if you don't try other machines due to randomness. But if you try other machines, you might lose an opportunity to earn more money. What should I do? This is a well-known Exploration and Exploitation trade-off in RL.
RL trying to maximize the gain including the gains right now and the gains in the future. In other words, the performance during training is also considered as its performance. That's why RL is not unsupervised nor supervised learning.
However, in some situations, you might want to separate training and testing. RL is designed for an agent who interacts with the environment. However, in some cases, (for example), rather than having an interactive playground, you have data of interactions. The formulation would be a little different in this case.
