I am reading AI: A Modern Approach. In the 2nd chapter when introducing different agent types, i.e., reflex, utility-based, goal-based, and learning agents, I understood that all types of agents, except learning agents, receive feedback and choose actions using the performance measure.
But they do so in different ways. Model-based reflex agents possess an internal state (like a memory), while goal-based agents predict the outcome of actions and choose the one serving the goal. Lastly, utility-based functions measure the 'happiness' of each state using the utility function, which is again an internalization of the performance measure, hence all have similar nature overall.
The learning agents, however, can be wrapped around the entire structure of previous agents. The entire agent's architecture is now called a performance element, and the learning agent has an additional learning element, which modifies each component of the agent, so as to bring the components into closer agreement with the available feedback information. But the feedback information in learning agents does not from the performance measure embedded in the agent's structure, but from a fixed external performance standard, which is part of the critic element*.
For the purpose of illustration, the structure of a utility-based agent and that of a learning agent are presented in the figure:
What boggles my mind is figuring out the actual difference and interaction between performance standard and performance measure, which is perhaps related to those between learning agents and other ones. Here are my thoughts thus far:
Other agents aim for maximizing the performance measure, causing them to do perfect actions. On the other hand, learning agents have the freedom of doing sub-optimal actions, which allow them to discover better actions on the long run using the performance standard.
Through the performance standard's feedback (which comes from the critic as shown in the figure), the learning agent can also learn a utility function or reflex component.
For providing examples, the book states that giving tip to an automated taxi is considered a performance standard. And also
hard-wired performance standards such as pain and hunger in animals can be understood in this way.
But I am still not sure about the discrepancy and interaction between the performance measure and performance standard. For instance, in the automated taxi, when confronting a road junction, the utility-based agent chooses a path that maximizes its utility function. The learning agent, however, must check different roads and after testing them, it receives feedback from outside so that eventually it would detect the user's preference.
But what if we wrap a learning agent around a utility-based agent in such a condition? Which has more effect, the utility function from inside, or the performance standard from outside (critic)? If they happen to contradict each other, which one would have the prevalent effect?