from what I understand you are building you own model for this specific use case. From my perspective I would try not to reinvent the wheel, as it is said, and use an already proven and working model such as the YOLOs (v1, v2 and v3).
YOLO does not tell you the center pixel of the image directly but it tells you the center cell, with respect to the predicted object, of a grid (which is built on top of the image) responsible for predicting each object. See on the left image how the grid is built on top of the input image, then YOLO computes a probability map, and then the bounding boxes are predicted from the cell in the object's center. I have highlighted which cells would be responsible for predicting each object in the rightmost image. (Image from YOLOv1 paper)
The grid can have different resolutions but if you make it equal to the image size, then you would have the center pixel of each object predicted. This is because YOLO predicts objects in each cell of the grid, so if the grid is equal to the image size, YOLO will predict objects in each pixel.
As an example, imagine you have an input image of $[H \times W] = [416 \times 416]$ then YOLO would compute a grid of $[S_1 \times S_2]=[52 \times 52]$ on top of it. And predict objects in the center cell of the $[S_1 \times S_2]$. So, if you tune YOLO for computing a grid such as $[S_1 \times S_2] = [H \times W]$, then YOLO would output objects prediction with respect to the image pixels, in other words, YOLO would predict bounding boxes centered in the image pixel on the object's center.
This is how I would proceed for this use case, I hope it helps you or at least give you some clues about how to proceed further! Cheers! :)
NOTE: I chose the image size and grid size with numbers I usually see at work. Specifically, using YOLOv3. In YOLOv3, for aninput image of $[H \times W] = [416 \times 416]$, 3 grids are built, with different resolutions (for predicting big and small objects), with the following grids sizes are: $[13 \times 13], [26 \times 26], [52 \times 52]$