I want to train a neural network on some input data from a probability distribution (say a Gaussian). The loss function would normally be $-\sum\log(f(x_i))$, where the sum is over the whole data (or in this case a mini batch) and $f$ is the NN function. However I need to enforce the fact that $\int_0^\infty f(x)dx=1$, in order for $f$ to be a real probability distribution. How can I add that to the loss function? Thank you!

  • $\begingroup$ The loss you mentioned is used for multi class classification. Is your problem classification or regression? $\endgroup$
    – user9947
    Jun 8 '19 at 4:19
  • $\begingroup$ I am given data points generated from a function (distribution) as input, and I want to train a NN to approximate that function. I am not sure if it is classification or regression. I just want to train the network to approximate f. $\endgroup$ Jun 8 '19 at 5:00
  • $\begingroup$ But apparently the function is a probability distribution. The final output will be taken from a single node I guess? $\endgroup$
    – user9947
    Jun 8 '19 at 5:29
  • $\begingroup$ Both the input and the output are one single node, yes. It can be shown mathematically that given data generated by a probability distribution, the function that generated the data is the one minimizing the loss function I wrote there. I just need to impose that normalization (otherwise the constant function equal to 1 would give the smallest loss). But I am not sure how to do that. $\endgroup$ Jun 8 '19 at 5:54
  • $\begingroup$ I assume the inputs are continuous? $\endgroup$
    – user9947
    Jun 8 '19 at 6:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.