How can I simulate responses from the distribution of human intelligence?

Question

I'm eventually looking to build an algorithm that will process answers from humans that are given questions. But first I have to setup an experiment to determine the variety of responses.

Specifically, humans will be asked a multiple choice question that has a single correct answer. I want to understand what kinds/ranges of responses I would get from the bell curve distribution of human intelligence.

Is there any way I can have, say, 1000 "humans" be asked a prompt, repeated 100 times (the same question) and then compile the responses? My concern is that I'll have to build some algorithm or process for each dumb, average, smart "human" to follow but then I would introduce bias in how smart they are or limit how they may respond. I'm guessing I'll have to give them a data sort to work from.

To clarify, it's not the number of times a single user gets a question right that makes them smart, they have to be programmed dumb, smart etc. before the simulation starts. So dumb users could get some right and smart can get some wrong.

I'm not sure the Monte Carlo method is useful here but some type of simulation where I can specify the distribution (normal) and then bound the responses would be helpful.

I have access to Excel, Minitab, and Python. Any ideas how to set up an experiment like this? I really am open to any technique to measure this.

Answer 1

The data you want to collect cannot be reliably simulated at this time. There is no current realistic simulator for a human performing reading comprehension.

The actual error rates and specific wrong answers chosen on the questions will depend to a large degree on specific humans, and the nature of questions. As you are hoping to get realistic results, the only method that will work for your ground truth data is to present 1000+ real humans with your sample questions. In addition, if you want to categorise your humans into "smart", "dumb" etc, you will need to run additional tests on them, such as an IQ test, in order to create those categories.

Depending on context, such as the nature of the questions you want to assess, you may be able to obtain some anonymised data from real-world exams that could help, instead of trying to generate that data yourself. In that case you could make an approximate model for humans answering multiple-choice questions - perhaps by training an LSTM-based natural language model. For best accuracy on your questions you would want the training set to include similar kinds of question. There is still a caveat that NNs do not really do logic or reasoning, they make statistical fits, so can easily get answers logically wrong or select nonsense answers. The best general NLP models still fail badly at semantics.

If you have specific set of questions to evaluate - and are not willing to ignore the content of the questions - then no machine can currently match distribution of human behaviour on such a task without extensive training data and significant effort.

If you don't care about the content of the questions, or assessing their difficulty, or indeed any feedback about the specific questions, then you could maybe use a dataset from any multi-choice questionnaire to get statistics of correct answer accuracy. With this approach your simulation could just be a simple distribution over "correct", "most plausible incorrect" etc answers where you choose "most plausible incorrect" either manually or randomly (but consistently). This will get you a distribution of responses similar to known real-world data. It could be used to unit-test a scoring system perhaps, or demonstrate some stats or visualisation software. But the question and answer text may as well be gibberish at that point.