Sutton & Barto: what are parametrized functions?

Question

From "Reinforcement Learning: An introduction (2nd ed.)" by Richard S. Sutton and Andrew G. Barto, on page 59

Instead, the agent would have to maintain $v_\pi$ and $q_\pi$ as parameterized functions (with fewer parameters than states) and adjust the parameters to better match the observed returns.

What do they mean by "parametrized functions" here? The part of "adjust parameters to better match observed returns" also sounds odd to me.

Answer 1

A parameterized function is a function that is defined by a set of parameters. If you change the parameters, you also change the actual function. For example, let's define this linear function

$$f: \mathcal{X} \rightarrow \mathcal{Y}$$ as follows

$$f(x; a, b) = ax + b \in \mathcal{Y}.$$

Here, $x \in \mathcal{X}$ is the input to $f$, while $a \in \mathbb{R}$ and $b \in \mathbb{R}$ are the parameters. Clearly, if you change $a$ and $b$, you actual get different functions. If that's not obvious, you can graph them with a tool (e.g. this one). So, a parametrized function can actually be thought of a set of functions or a model. For more details about how functions and models are related in AI, see this answer.

Now, back to the specific context. $v_\pi$ and $q_\pi$ are value functions, so they give you the value of a state or state-action pair, respectively, i.e. the expected return (reward in the long run). The goal in RL is really to learn these functions or a policy. Now, Sutton & Barto mean that we can use a model like $f$ above and adjust the parameters $a$ and $b$ in order to learn the optimal value functions. Now, I will not define what an optimal value function is. It's defined in the book. Once you have a good grasp of the notion of a value function and optimality, you can also check this post (written by me) for more mathematical details on Bellman equations and value functions.

Now, why would you want to use a function like $f$ to approximate these value functions? The same book tells you why in the sentence before - for finite MDPs, we could use an array to represent these functions, but if there are many states, actions, etc., it becomes hard to learn these functions in this way. If we use parameterized functions, we do not need to maintain so many entries (one for each state or state-action pair) and we hope that the function can generalise to unseen or rarely seen inputs.

Finally, in practice, people have been using neural networks to learn value functions. The difference between neural networks and $f$ above is that neural networks are non-linear, more powerful, but they also provide fewer guarantees.