2
$\begingroup$

Let's say I have an adjustable loaded die, and I want to train a neural network to give me the probability of each face, depending on the settings of the loaded die.

I can't mesure its performance on individual die roll, since it does not give me a probability.

I could batch a lot of roll to calculate a probability and use this batch as an individual test case, but my problem does not allow this (let's say the settings are complex and randomized between each roll).

I have 2 ideas:

  1. train it as a classification problem which output confidence, and hope that the confidence will reflect the actual probability. Sometimes the network would output the correct probability and fail the test, but on average it would tend to the correct probability. However it may require a lot of training and data.
  2. batch random rolls together and compare the mean/median/standard deviation of the measured result vs the predictions. It could work but I don't know the good batch size.

Thank you.

$\endgroup$
  • 1
    $\begingroup$ The idea of a Neural net is to approximate a function. We do that by having a train and test set, any NN can generalize on the training set. The problem of a NN is generalizing on test set which is possible iff the NN is generalizing a non random function of the input variables. Now think whether your problem matches this setting. $\endgroup$ – DuttaA Jul 3 '19 at 19:27
  • $\begingroup$ @DuttaA while a function cannot predict a random ouput, It could theorically predict probabilities, which are not random. either I need to built test cases by grouping individual entries into a statistical distribution, or find another clever method. $\endgroup$ – Steve Marion Jul 3 '19 at 19:33
  • $\begingroup$ predicting probablities is not so easy, you have to make choice for CDF model or PDF model. Then if you have a loaded die then why cannot normal mechanics approach be used to predict it's falling state for a particular starting state? (Might be difficult tho) $\endgroup$ – DuttaA Jul 3 '19 at 19:37
0
$\begingroup$

Neural network isnt what you want here. You have a limited number of events and draws from some unknown distribution that you want to recover.

In that case, just use the empirical probabilities $p(event_i) = \frac{\# event_i}{total\ events}$ which given enough draws will converge to the true probability.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I'm not sure I can do that because there are too many variables (up 40 dimensions with value range 1-100) in input, I don't know how these properties relate and an empirical approach would require too much data. But it could be a valid approach if I make some assumptions on the model to simplify it. $\endgroup$ – Steve Marion Jul 3 '19 at 19:25
  • 2
    $\begingroup$ If you cannot take an empirical approach how will you generate training data for your ML algorithm? $\endgroup$ – DrMcCleod Jul 3 '19 at 20:05
0
$\begingroup$

This approach:

train it as a classification problem which output confidence, and hope that the confidence will reflect the actual probability. Sometimes the network would output the correct probability and fail the test, but on average it would tend to the correct probability.

will work with some limitations. If you use a classifier with softmax activation and multiclass log loss

$$\mathcal{L}(\mathbf{\hat{y}},\mathbf{y}) = -\mathbf{y} \cdot \text{log}(\mathbf{\hat{y}})$$

where $\mathbf{\hat{y}}$ is the network output as a vector, and $\mathbf{y}$ is the actual output from an individual sample. Your input should be the settings of the die.

Optimising this loss will converge on approximated probabilities for each discrete output. You can demonstrate this with some simple examples - for instance if you train a network with a single input - a one-hot-encoded die type from the classic D&D dice sets, plus deliberately chosen examples of different results in the right frequencies, you will end up with a classifier that predicts roughly $p=0.25$ for results of 1,2,3,4 for a d4 and $p=0.125$ for results of 1,2,3,4,5,6,7,8 for a d8

So it works mathematically. Whether it works for your situation depends on details. You need enough data samples to cover both the distribution of results under each setting, and any complexities of how that distribution varies with the settings. In the limit of wanting very accurate predictions of probability within a complex space you will need a huge number of samples. You should be able to find a compromise between accuracy and generalisation by trying different levels of regularisation - this will be necessary as over-fitting to input/output sample pairs as seen is going to be a serious problem for a neural network trained on this data.

One thing you can do to help a classifier learn probabilities is always take some number of samples with the same settings - e.g. 10 or 100 or 1000 each with the same settings - that should guarantee that the network cannot simply converge to predict high $p$ values for single outputs as seen, as it will have counter-examples to work with.

You mention that you have 40 dimensions of settings. Whether this is an issue will depend on how the probability distribution varies based on those settings. However, at minimum you should be thinking in terms of millions of samples for training, or possibly a fast on-demand generator that can generate 1000s of new samples per second to train with.

You can test accuracy by building histograms using some fixed (and as yet unseen) setting and comparing to NN predictions of probabilities of that setting. Even getting accurate test results is likely to require 1000s of samples.

However it may require a lot of training and data.

If you cannot obtain a very large training set here, then a purely statistical "black box" approach is probably not feasible, regardless of whether you use neural networks, or more raw analysis. What neural networks add is smooth interpolation between different settings values, as a form of approximation. This seems desirable for your problem, as you will never fully explore 40 dimensions of 100 values in the lifetime of the universe - but you need some confidence that minor changes in settings equate to minor changes in probability distributions in most cases.

It's OK to have one or two major shifts across the input space, but if the distribution depends in some cryptographic primitive or similar complex high frequency (over space) and high amplitude on the input variables, there is no way to obtain approximations using statistics.

The alternative to statistical approaches is to find some way to break open this black box through analysis. No AI system can do that in general at the moment, so you would rely on human ingenuity.

| improve this answer | |
$\endgroup$
  • $\begingroup$ while I have a lot of input variables, I know there is some underlying rule that simplify a lot the problem (ie. reduce the actual number of dimension of the input). if I apply a statistical analysis I would need to make assumption on the rules to simplify beforehand. I would rather the NN to infer it from its training. I can generate 1-2 tests/s. what about the 2nd approach of evaluating the loss on a group of test cases ? $\endgroup$ – Steve Marion Jul 4 '19 at 13:03
  • $\begingroup$ @SteveMarion: In that case it may be tractable. You will still need a lot of training data for the network to learn about this embedded lower-dimension problem. $\endgroup$ – Neil Slater Jul 4 '19 at 13:07
  • $\begingroup$ @SteveMarion: A NN is a statistical method. It "infers" data that is available statistically. You don't need to pre-analyse anything before feeding into a NN, and I don't suggest you need to in this answer. With 1-2 tests/second it should take you about 2 weeks to generate a dataset of a million records. Assuming that is automatic and not a manual process, that seems quite a reasonable time investment for data gathering. Of course that number is a guess and there are ways to check whether more data would be more useful before making the effort $\endgroup$ – Neil Slater Jul 4 '19 at 13:11
0
$\begingroup$

The starting point is that for a fair dice thrown fairly the p(n) is 1/n where n is the number of sides.

You said both

and

there are too many variables (up 40 dimensions with value range 1-100) in input, I don't know how these properties relate and an empirical approach would require too much data.

It seems that this problem has 2 solutions:

  1. Uon't use a neural net and create a 'std' statistical model. It may be possible since you said:

    I know there is some underlying rule that simplify a lot the problem (ie. reduce the actual number of dimension of the input)

  2. Use a neural network (with softmax at the end) - for a fair dice; with enough training data the classifier should arrive as 1/n as the approximating function for a fair dice. The other 40 dimensions/settings your mentioned are the inputs. I think a 'basic' neural network with dense layers only could work for your task.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.