Let's say that I used a keypoint detector like SIFT or SuperPoint to detect keypoints in image 1 and 2. Afterwards, I used a keypoint matcher to match corresponding keypoints in this image pair. The first screenshot below shows the zoomed-in patches around a matched keypoint pair (keypoints are incidated as green dots). You can see that the detected keypoints do not accurately correspond to the same real-world point. I would like to shift one of these keypoints to increase the matching accuracy. What kind of algorithms exist for this purpose?
I've tried to use a CNN to extract feature maps of image 1 and 2, and subsequently convolving the features of keypoint 1 (which has dimensionality $1\times1\times D$ with the feature maps of image 2 (dimensionality $W \times H \times D$), resulting in a kind of probability heatmap indicating where the keypoint of image 2 should be shifted to (see screenshot 2). Sometimes this works, but often it does not (see screenshot 3). Would adding attentional layers help with this problem? Or are there any known algorithms for this purpose?