How can I train a neural network to give probability of a random event?

Question

Let's say I have an adjustable loaded die, and I want to train a neural network to give me the probability of each face, depending on the settings of the loaded die.

I can't mesure its performance on individual die roll, since it does not give me a probability.

I could batch a lot of roll to calculate a probability and use this batch as an individual test case, but my problem does not allow this (let's say the settings are complex and randomized between each roll).

I have 2 ideas:

1. train it as a classification problem which output confidence, and hope that the confidence will reflect the actual probability. Sometimes the network would output the correct probability and fail the test, but on average it would tend to the correct probability. However it may require a lot of training and data.
2. batch random rolls together and compare the mean/median/standard deviation of the measured result vs the predictions. It could work but I don't know the good batch size.

Thank you.

Answer 1

The starting point is that for a fair dice thrown fairly the p(n) is 1/n where n is the number of sides.

You said both

and

there are too many variables (up 40 dimensions with value range 1-100) in input, I don't know how these properties relate and an empirical approach would require too much data.

It seems that this problem has 2 solutions:

1. Don't use a neural net and create a 'std' statistical model. It may be possible since you said:

I know there is some underlying rule that simplify a lot the problem (ie. reduce the actual number of dimension of the input)

1. Use a neural network (with softmax at the end) - for a fair dice; with enough training data the classifier should arrive as 1/n as the approximating function for a fair dice. The other 40 dimensions/settings your mentioned are the inputs. I think a 'basic' neural network with dense layers only could work for your task.

Answer 2

Neural network isnt what you want here. You have a limited number of events and draws from some unknown distribution that you want to recover.

In that case, just use the empirical probabilities $p(event_i) = \frac{\# event_i}{total\ events}$ which given enough draws will converge to the true probability.

Answer 3

This approach:

train it as a classification problem which output confidence, and hope that the confidence will reflect the actual probability. Sometimes the network would output the correct probability and fail the test, but on average it would tend to the correct probability.

will work with some limitations. If you use a classifier with softmax activation and multiclass log loss

$$\mathcal{L}(\mathbf{\hat{y}},\mathbf{y}) = -\mathbf{y} \cdot \text{log}(\mathbf{\hat{y}})$$

where $\mathbf{\hat{y}}$ is the network output as a vector, and $\mathbf{y}$ is the actual output from an individual sample. Your input should be the settings of the die.

Optimising this loss will converge on approximated probabilities for each discrete output. You can demonstrate this with some simple examples - for instance if you train a network with a single input - a one-hot-encoded die type from the classic D&D dice sets, plus deliberately chosen examples of different results in the right frequencies, you will end up with a classifier that predicts roughly $p=0.25$ for results of 1,2,3,4 for a d4 and $p=0.125$ for results of 1,2,3,4,5,6,7,8 for a d8

So it works mathematically. Whether it works for your situation depends on details. You need enough data samples to cover both the distribution of results under each setting, and any complexities of how that distribution varies with the settings. In the limit of wanting very accurate predictions of probability within a complex space you will need a huge number of samples. You should be able to find a compromise between accuracy and generalisation by trying different levels of regularisation - this will be necessary as over-fitting to input/output sample pairs as seen is going to be a serious problem for a neural network trained on this data.

One thing you can do to help a classifier learn probabilities is always take some number of samples with the same settings - e.g. 10 or 100 or 1000 each with the same settings - that should guarantee that the network cannot simply converge to predict high $p$ values for single outputs as seen, as it will have counter-examples to work with.

You mention that you have 40 dimensions of settings. Whether this is an issue will depend on how the probability distribution varies based on those settings. However, at minimum you should be thinking in terms of millions of samples for training, or possibly a fast on-demand generator that can generate 1000s of new samples per second to train with.

You can test accuracy by building histograms using some fixed (and as yet unseen) setting and comparing to NN predictions of probabilities of that setting. Even getting accurate test results is likely to require 1000s of samples.

However it may require a lot of training and data.

If you cannot obtain a very large training set here, then a purely statistical "black box" approach is probably not feasible, regardless of whether you use neural networks, or more raw analysis. What neural networks add is smooth interpolation between different settings values, as a form of approximation. This seems desirable for your problem, as you will never fully explore 40 dimensions of 100 values in the lifetime of the universe - but you need some confidence that minor changes in settings equate to minor changes in probability distributions in most cases.

It's OK to have one or two major shifts across the input space, but if the distribution depends in some cryptographic primitive or similar complex high frequency (over space) and high amplitude on the input variables, there is no way to obtain approximations using statistics.

The alternative to statistical approaches is to find some way to break open this black box through analysis. No AI system can do that in general at the moment, so you would rely on human ingenuity.