I'm interested in learning to rank with pairwise comparison. While working on this, I found that XGBoost has a model called XGBRanker, which works very well.
I want to find out how the XGBRanker manages the training data to get such low memory usage and great results. (It uses LambdaMART I believe).
I imagine it must be some kind of lookup table for the features and maybe making the pairs iteratively or not using all possible permutations with different labels within one group.
I tried looking through the source code, but everything keeps referring to some other XGBoost method and I haven't been able to understand it so far.
I would like to create a similar method to train NNs for pairwise comparison, but handling the training data has been a huge hurdle so far.
So, more generally, my question would be: How are the pairs created in pairwise ranking algorithms (RankNet, LambdaNet, and so on)? Are all pairs used? Only a percentage? Is there some other way of doing this? If you're working with >100.000 items, you would easily get into the range of hundreds of millions.