I've always found bounding box regression a bit weird. There's no positional encoding like in vision transformers, so how does the network "know" the absolute position when producing bounding box coordinates? It gets even weirder when we are dealing with two-stage detectors because, in the second stage bounding box regression, only an ROI is available, not the whole image.
It of course depends on the detection model that is used.
But in your case I think you relate to a Faster-RCNN type architecture for bounding box detection. In this case only the relative position to an anchor is regressed, that is correct. Of course the regression "values" isolated have no information where they are absolute in the image since the same convolution is shared and slided over the whole image/layer. However, after your regression layer is done (i.e. slided the convolution over the whole previous layer) it outputs a new feature map with relative regression values. The regression values itself do not contain the information about the abolute position, but the xy position of the relative feature values inside the output feature map of the regressor combined with the relative values itself gives you the information about the absolute position.
This also makes it "translation invariant", which is an important and wanted feature of Faster-RCNN architectures.