# How to properly evaluate RL agents while varying evaluation environments?

I am using Stable-Baselines and my goal is to have agents rank a list of items (or rather, put the relevant items on top), similarly to what learning to rank algorithms (e.g. lambdarank) do as part of the information retrieval domain.

I created a pairwise comparison environment, basically, it performs a merge sort on the list of items, and relies on the agent to do the comparisons. The observation space consists of two arrays, each containing the features of an item, and the action space is $$\{0, 1 \}$$ ($$0$$ = first item should be ranked higher, $$1$$ = second item should be ranked higher). The reward is $$1$$ if the selected item is indeed relevant or if both items are, $$-1$$ if the selected item is not relevant while the unselected one is, and $$0$$ if neither is relevant.

The list of items varies over time (new items, some items removed). So, the agent learns from the past versions of the list, and then predicts which items are relevant on the current version of the list. Hence, this is an iterative process, the agent is trained every time there's a new version of the list.

To periodically evaluate the agent and save its best version, I use a set of past versions of the list. But I need this set to change over time, swapping lists with others, as it makes more sense to evaluate the agent on versions of the list that are as closely related to the current list as possible.

My question is: How to properly evaluate agents when the evaluation set of lists fluctuates?

If the lists are not of the same size, then the evaluation episodes will have a different number of steps ($$n \log n$$ steps of a list of $$n$$ items). It means an unequal number of opportunities for the agent to be rewarded. Also, the more relevant items in the list, the more opportunities for positive AND negative rewards.

In stable-baselines, two agents are compared based on their evaluation score, which is the mean total reward. I can't do that in my situation, since the best agent was evaluated on a past set of lists. I need to account for the number of items and the number of relevant items.

I could always reevaluate the best agent on the new set, but this could become time-consuming in the long run, especially if the list is very large. So, I was wondering if it was possible to find a formula that would factor in the list size and number of relevant items. Does anybody have an idea?

Any assistance you can provide would be greatly appreciated.