I'm eventually looking to build an algorithm that will process answers from humans that are given questions. But first I have to setup an experiment to determine the variety of responses.
Specifically, humans will be asked a multiple choice question that has a single correct answer. I want to understand what kinds/ranges of responses I would get from the bell curve distribution of human intelligence.
Is there any way I can have, say, 1000 "humans" be asked a prompt, repeated 100 times (the same question) and then compile the responses? My concern is that I'll have to build some algorithm or process for each dumb, average, smart "human" to follow but then I would introduce bias in how smart they are or limit how they may respond. I'm guessing I'll have to give them a data sort to work from.
To clarify, it's not the number of times a single user gets a question right that makes them smart, they have to be programmed dumb, smart etc. before the simulation starts. So dumb users could get some right and smart can get some wrong.
I'm not sure the Monte Carlo method is useful here but some type of simulation where I can specify the distribution (normal) and then bound the responses would be helpful.
I have access to Excel, Minitab, and Python. Any ideas how to set up an experiment like this? I really am open to any technique to measure this.