I have a huge dataset of 3D point clouds (each point consists of X,Y,Z coordinates) and another dataset with keypoints (also X,Y,Z) which lie on quite recognizable structures in the point cloud. As a human it is pretty easy to find these keypoint given the according point cloud.
There are 6-24 keypoints for each point cloud which may have up to 100.000 points but can be downsampled to fit into memory while processing.
My goal is to train a supervised deep learning model to detect/generate these keypoints in new point clouds. These have to be new synthetic points since the keypoints are most likely not in my point cloud.
Previously i tried to train a regression model using PointNet++ semantic segmentation. I generated weights for each point in the input point cloud according to its proximity to the next keypoint. For new point clouds the weights were predicted pretty well but it is very difficult to regenerate the synthetic keypoints from these weights since the area around the keypoint can be very sparse.
Essentially i am trying to use a model like this RSN on 3D point clouds to detect synthetic keypoints in the data. I thought about using object detection models like VoxelNet but these detect bounding boxes and i want to get a single predicted point as a result.
I am also wondering if its worth looking into GANs but i dont have experience with these models and until now i only found models who generate a whole new point cloud instead of a single points inside a given point cloud.
I would love to hear some ideas from you to point me in the right direction.