What is the difference between a performance standard and performance measure?

Question

I am reading AI: A Modern Approach. In the 2nd chapter when introducing different agent types, i.e., reflex, utility-based, goal-based, and learning agents, I understood that all types of agents, except learning agents, receive feedback and choose actions using the performance measure.

But they do so in different ways. Model-based reflex agents possess an internal state (like a memory), while goal-based agents predict the outcome of actions and choose the one serving the goal. Lastly, utility-based functions measure the 'happiness' of each state using the utility function, which is again an internalization of the performance measure, hence all have similar nature overall.

The learning agents, however, can be wrapped around the entire structure of previous agents. The entire agent's architecture is now called a performance element, and the learning agent has an additional learning element, which modifies each component of the agent, so as to bring the components into closer agreement with the available feedback information. But the feedback information in learning agents does not from the performance measure embedded in the agent's structure, but from a fixed external performance standard, which is part of the critic element*.

For the purpose of illustration, the structure of a utility-based agent and that of a learning agent are presented in the figure:

What boggles my mind is figuring out the actual difference and interaction between performance standard and performance measure, which is perhaps related to those between learning agents and other ones. Here are my thoughts thus far:

1. Other agents aim for maximizing the performance measure, causing them to do perfect actions. On the other hand, learning agents have the freedom of doing sub-optimal actions, which allow them to discover better actions on the long run using the performance standard.

2. Through the performance standard's feedback (which comes from the critic as shown in the figure), the learning agent can also learn a utility function or reflex component.

For providing examples, the book states that giving tip to an automated taxi is considered a performance standard. And also

hard-wired performance standards such as pain and hunger in animals can be understood in this way.

But I am still not sure about the discrepancy and interaction between the performance measure and performance standard. For instance, in the automated taxi, when confronting a road junction, the utility-based agent chooses a path that maximizes its utility function. The learning agent, however, must check different roads and after testing them, it receives feedback from outside so that eventually it would detect the user's preference.

But what if we wrap a learning agent around a utility-based agent in such a condition? Which has more effect, the utility function from inside, or the performance standard from outside (critic)? If they happen to contradict each other, which one would have the prevalent effect?

Answer 1

After having read a few parts of the book that mention these terms, I don't think there's any practical difference between a performance standard and a performance measure. They both measure the performance of the agents.

They initially use the term performance measure to refer to rationality in section 1.1

The definitions on the left measure success in terms of fidelity to human performance, whereas the ones on the right measure against an ideal performance measure, called rationality.

Later, they define this performance measure in the context of rational agents in section 2.2.

If the sequence is desirable, then the agent has performed well. This notion of desirability is captured by a performance measure that evaluates any given sequence of environment states.

So, here, a performance measure evaluates a sequence of states.

In that same section (3rd edition), they explain that the performance measure should be designed in such a way that the agent achieves our desired goals.

Obviously, there is not one fixed performance measure for all tasks and agents; typically, a designer will devise one appropriate to the circumstances. This is not as easy as it sounds. Consider, for example, the vacuum-cleaner agent from the preceding section. We might propose to measure performance by the amount of dirt cleaned up in a single eight-hour shift. With a rational agent, of course, what you ask for is what you get. A rational agent can maximize this performance measure by cleaning up the dirt, then dumping it all on the floor, then cleaning it up again, and so on. A more suitable performance measure would reward the agent for having a clean floor. For example, one point could be awarded for each clean square at each time step (perhaps with a penalty for electricity consumed and noise generated). As a general rule, it is better to design performance measures according to what one actually wants in the environment, rather than according to how one thinks the agent should behave.

So, the usage of the term performance measure in this example is very analogous to the usage of the term reward function in RL. To be more precise, in RL, you could define a reward function as a function $r : \mathcal{S} \rightarrow \mathbb{R}$, so $r(s)$, where $s \in \mathcal{S}$, is the reward given to the RL agent when it enters or is in state $s$. This is consistent with their definition above (although they state that the performance measure evaluates a sequence of states, but I think this is due to the fact that, in a rational agent, you don't necessarily assume that the Markov property holds, like in an MDP and RL).

Later, they also use the term performance measure in the context of utility-based agents,

An agent's utility function is essentially an internalization of the performance measure. If the internal utility function and the external performance measure are in agreement, then an agent that chooses actions to maximize its utility will be rational according to the external performance measure

So, here, if you are familiar with RL, the distinction between the utility function and the performance measure should be clear. The utility function would be analogous to the value function, while the performance measure is again analogous to the reward function.

Later, in section 17.1.1, they write

In the MDP example in Figure 17.1, the performance of the agent was measured by a sum of rewards for the states visited. This choice of performance measure is not arbitrary

So, again, the performance measure is a synonym for the reward function in the context of RL.

In section 2.4, they write about the performance standard

The critic tells the learning element how well the agent is doing with respect to a fixed performance standard. The critic is necessary because the percepts themselves provide no indication of the agent’s success. For example, a chess program could receive a percept indicating that it has checkmated its opponent, but it needs a performance standard to know that this is a good thing; the percept itself does not say so. It is important that the performance standard be fixed. Conceptually, one should think of it as being outside the agent altogether because the agent must not modify it to fit its own behavior

That's why it appears outside the agent in the diagram.

On the next page, they write

In a sense, the performance standard distinguishes part of the incoming percept as a reward (or penalty) that provides direct feedback on the quality of the agent’s behavior.

In RL, this would be represented as $p = (o, r)$, i.e. the percept $p$ is composed of the observation $o$ and reward/performance $r$.

I don't really know why they used different terms for rational, utility-based, and learning agents, but, to me, I don't see any difference between the two terms.