I don't understand the difference between a policy and rewards. Sure, a policy tells us what to do, but isn't the output of a neural network trained on rewards basically a policy (i.e. choose the maximum reward)? What is different about the policy? An extra softmax applied?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
8
A (stochastic) policy is a set of conditional probability distributions, $$\pi(a \mid S=s), \forall s \in \mathcal{S}.$$ If the policy is deterministic, then it is a function $$\pi: \mathcal{S} \rightarrow \mathcal{A},$$ so $\pi(s) = a$ is the action that policy $a$ returns in the state $s$ - it always produces this same action for a given state, unless it's a non-stationary policy. A policy is also called a strategy (in game theory). To be usable, a stochastic policy must be turned into a decision rule, i.e. you need to sample from it. A stochastic policy generalises a deterministic one.
The rewards are the outputs of the reward function. A reward function can be deterministic (which is often the case) or stochastic. It can be defined as $$R : \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{R},$$ where $\mathcal{R} \in \mathbb{R}$ is the reward space. If it's stochastic, then
\begin{align} R(s, a) &= \mathbb{E}\left[ R_t \mid S_t = s, A_t = a\right] \\ &=\sum_r r p(r \mid S_t = a, A_t=a), \end{align} where $R_t$ is the random variable that represents the reward at time step $t$ and assuming a finite MDP. Stochastic reward functions generalise deterministic ones.
So, policies are probability distributions or functions, while rewards are numbers. So, there's a difference between their definitions, even though they are related.
How are they related? In different ways. The most important one is that an optimal policy for a given MDP is the one that, if followed, maximises the expected return, which is a function of the reward (typically, a discounted sum of rewards). The definition of an optimal policy makes more sense if you also know the definition of a value function - I recommend you take Sutton & Barto's book and read the relevant sections.
-
$\begingroup$ Detailed answer, thanks. But I still don't get how these two things are different under the hood, meaning the neural network output. $\endgroup$ Apr 20, 2022 at 17:35
-
$\begingroup$ In deep RL, the neural network can represent the policy. To find the optimal policy, you need to explore the environment and collect rewards and understand which actions lead to more rewards. You then use these rewards to update the neural network, so the policy. So, the rewards are used to train the neural network. Maybe you should look at the equations of Q-learning or any other RL algorithm. In them, $Q(s, a)$ is the neural network, while $r$ is the reward. There's clearly a difference. One is a function, the other is a number, as I say in my answer. $\endgroup$– nbroApr 20, 2022 at 17:37
-
$\begingroup$ I recommend that you read the first chapters of the book mentioned in my answer. I think you don't understand the difference because you're missing some details, but I don't know which ones. Let me give you this example - hopefully, you're more familiar with supervised learning. Do you think that labels (e.g. dog or cat) are the same thing as a neural network that classifies dogs and cats? Here, the labels would be analogous to the rewards, while the classifier would be analogous to the policy. See this. $\endgroup$– nbroApr 20, 2022 at 17:40
-
$\begingroup$ Maybe I'm confused because a lot of papers, articles etc. refer to them as distinct things. What you're telling me I think is that the policy is the method (nn or other) and the reward is how we get to the method (which could also be value+advantage or something else). $\endgroup$ Apr 20, 2022 at 17:47
-
$\begingroup$ I wouldn't say that a policy is a method, if you're using it as a synonym for algorithm, although a decision rule can be thought of as an algorithm. Maybe you meant model (and not method)? A neural network is essentially a model, i.e. it can be used to mimic something. That something is usually a function. Any (continuous) function. That's why neural networks are called function approximators, i.e. they approximate functions. In this case, the function that the neural network approximates is the policy. A policy is the solution to an optimisation problem defined in terms of an MDP. $\endgroup$– nbroApr 20, 2022 at 17:50
$\begingroup$
$\endgroup$
Typically, the answer to a control problem in reinforcement learning (RL), is "What is the policy that maximises total reward?".
In really simple scenarios, that you might study to understand RL basics, this can be so obvious that you could just search ahead and discover the correct action without really using RL.
However, sticking to the formal definitions in RL allows you to tackle harder problems, where it is not obvious how to decide what to do, or maybe even how to access the best rewards.
In the formalism of RL:
Reward is a real-valued signal received after each time step. For RL theory to be useful, the distrbution of reward values should be the same if start state and action are the same (you may also make the distribution depend on the next state, but that doesn't change the rule, it just means reward and next state will be correlated).
A policy is a function for action choice, it takes the state as input, and returns a distribution over all possible actions $\pi(a | s): \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R} = \mathbb{Pr}\{A_t=a|S_t=s\}$. It may be deterministic and directly return a chosen action $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$
These are clearly very different things. Often we are interested in finding a policy that maximises an aggregate of rewards, but that doesn't make the policy simply related to rewards. Think of a chess game - which piece should you move to win? There is no immediate reward, and the consequence for a good or bad move will happen much later. The relationship between the policy in an early game state, and the reward at the end (e.g. +1 for a win, -1 for a loss) is not at all clear.
Only in the very simplest of scenarios, where taking actions directly led to already-predictable rewards, could you use your idea of $\pi(s) = \text{argmax}_a r(s, a)$ where $r(s,a)$ was a reward function. For a start, this will not help you look ahead more than one time step. What if the best reward now was followed by a really bad reward?
RL has the toolkit you can use to decide how to offset immediate vs future gains, and also how to learn what to do when you don't already have a simple function $r(s,a)$ which tells you what is going to happen in advance.
answered Apr 20, 2022 at 17:10