In the original Faster R-CNN paper, the authors parameterized the box coordinates for regression under RPN. Below is the snippet of how they computed it:
Though this confuses me. They set the number of anchor boxes at 256. Though the number of ground-truth boxes (I'll call $n^*$) for each image is highly variable and is never equal to 256, unless by some coincidence. Same can be said for the number of predicted boxes ($n$). So how do they compute these numbers? Would there be $(4)(256)n$ predicted parameters and $(4)(256)n^*$ truth parameters? In the case that $n \neq n^*$, which is most definitely possible especially in early epochs, the loss cannot be computed as the lengths do not match.
What am I misunderstanding here? There must be something inferred from the article because the authors never touch on it again.