2
$\begingroup$

I am looking at a problem which can be distilled as follows: I have a phenomenon which can be modeled as a probability density function which is "messy" in that it sums to unity over its support but is somewhat jagged and spiky, and does not correspond to any particular textbook function. It takes considerable amounts of time to generate these experimental density functions, along with conditional data for machine learning, but I have them. I also have a crude model which runs quickly but performs poorly, i.e., generates poor quality density functions.

I would like to train a neural network to transform the crude estimated pdfs to something closer to the experimentally generated pdfs, if possible.

To investigate this, I've further reduced this to the most toy-like toy problem I can think of: Feeding a narrow, smooth (relatively narrow) normal curve into a 1D convolutional neural network, and trying to transform it to a similar narrow curve with a different mean. Both input and output have fine enough support (101 points) to be considered as a smooth pdf.

Here is the crux of the problem I think I have: I do not know what a good loss function is for this problem.

L1, L2 and similar losses are useless, given that once the non-zero parts of the pdfs are non-overlapping, it doesn't matter how far apart the means are, the loss remains the same.

I have been experimenting with Sinkhorn approximations to optimal transport, to properly capture the intuition of "distance" but somewhat surprisingly these have not been helpful either. I think part of the problem may be an (unavoidable?) numerical stability issue related to the support, but I would not stake hard money on that assumption.

(If support is at percentiles on the [0,1] it is quite instructive (and dismaying) to look at the sinkhorn loss for normal functions with the mean directly on a point of support, vs normal functions with the mean directly between two points of support.)

For a problem in this vein, are there any recommended loss functions (preferably supported by or easily implement in PyTorch) which might work better?

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Moderators, I waffled between asking this here (for the machine learning application) vs Cross Validated (for the statistics insight.) If it seems better posed there, let me know and I can remove and repost. $\endgroup$
    – Novak
    Jul 7, 2019 at 20:51
  • $\begingroup$ Hi. What is the "crude deterministic model" used for in your context? You also say "I would like to train a neural network to transform the crude estimate to the experimental data", but to transform to what? Maybe edit your question to specify (even though from the title it seems you want to convert a "messy probability distribution" to another probability distribution). $\endgroup$
    – nbro
    Jul 7, 2019 at 21:04
  • $\begingroup$ @nbro I'm not sure what you're asking. Yes, I want to feed in the crude estimated pdf (along with conditional data) and get out a substantially better estimate of the messy, experimentally generated pdf, if that's what you mean. $\endgroup$
    – Novak
    Jul 7, 2019 at 21:11
  • $\begingroup$ You say "I also have a crude deterministic model which runs quickly but performs poorly.". What does "crude deterministic model" mean in this context? Model of what? What is the relationship between this model and the probability density that models the phenomenon? $\endgroup$
    – nbro
    Jul 7, 2019 at 21:13
  • 1
    $\begingroup$ Ok, thanks for clarifying. $\endgroup$
    – nbro
    Jul 7, 2019 at 21:28

0

Reset to default

You must log in to answer this question.