I am looking at a problem which can be distilled as follows: I have a phenomenon which can be modeled as a probability density function which is "messy" in that it sums to unity over its support but is somewhat jagged and spiky, and does not correspond to any particular textbook function. It takes considerable amounts of time to generate these experimental density functions, along with conditional data for machine learning, but I have them. I also have a crude model which runs quickly but performs poorly, i.e., generates poor quality density functions.
I would like to train a neural network to transform the crude estimated pdfs to something closer to the experimentally generated pdfs, if possible.
To investigate this, I've further reduced this to the most toy-like toy problem I can think of: Feeding a narrow, smooth (relatively narrow) normal curve into a 1D convolutional neural network, and trying to transform it to a similar narrow curve with a different mean. Both input and output have fine enough support (101 points) to be considered as a smooth pdf.
Here is the crux of the problem I think I have: I do not know what a good loss function is for this problem.
L1, L2 and similar losses are useless, given that once the non-zero parts of the pdfs are non-overlapping, it doesn't matter how far apart the means are, the loss remains the same.
I have been experimenting with Sinkhorn approximations to optimal transport, to properly capture the intuition of "distance" but somewhat surprisingly these have not been helpful either. I think part of the problem may be an (unavoidable?) numerical stability issue related to the support, but I would not stake hard money on that assumption.
(If support is at percentiles on the [0,1] it is quite instructive (and dismaying) to look at the sinkhorn loss for normal functions with the mean directly on a point of support, vs normal functions with the mean directly between two points of support.)
For a problem in this vein, are there any recommended loss functions (preferably supported by or easily implement in PyTorch) which might work better?