4
$\begingroup$

I have a dataset of millions of chat messages from different discussions.

Some of the messages are written by people who lack understanding or relevant language skills. These messages almost always contain little to no thought and are almost always irrelevant to the main topic.

Some of the messages are written by highly educated people. The lexical structure of these messages is more complex.

What popular algorithm can yield results of good precision for this task - measuring or classifying human intelligence ?

Let's say the goal is classifying the data in these 3 categories: "dumb", "average", "intelligent".

The exact measuring of people's Intelligent Quotient by the way they write is a bit more interesting topic but I expect this would require far more research and fiddling.

I found almost no resources on prior research on this topic.

Has anyone come across a labeled dataset that might be useful for training a classifier on this task ?

$\endgroup$
  • 1
    $\begingroup$ words are not sufficient to tell if people are intelligent or not. I saw very dumb people using sophisticated words that you can thought they were intelligent (they just used these words wrong, without knowing their meanings) + "Some of the messages are written by people from the 3rd world. These messages almost always contain little to no thought and are almost always irrelevant to the main topic." is borderline.... $\endgroup$ – Jérémy Blain Oct 31 '18 at 16:20
  • $\begingroup$ However, they way sentences are constructed / the mutual dependecies between words can define well the intelligence of their author. $\endgroup$ – Boris D. Teoharov Oct 31 '18 at 16:43
0
$\begingroup$

No matter, if the score of a human or a chatbot has to be determined, the first thing to figure out is an evaluation measurement. That is a game, which the individual has to win. If many subjects are playing the same game, it is possible to determine their relative position on the scale. That means, the individual with the highest score in the game, gets the most points while the player who isn't playing well gets the lowest.

In the case of a repository of chat message such a game is missing. That means, the language has some meaning, and it is the result of a communication game, but there is no information what the game was about. In the literature the problem is known as grounding. Grounded speech means, that below the surface a formal game is available in which the communication process is the result.

To determine the score of an individual (only by parsing the chat logs) the first step is to recognize which type of game the individuals are playing. Let us make an example: if the resulting speech looks like a chat from the Scrabble game, it is possible to say who has won the game. That means, we have to parse only the speech act and can determine who is better than the opponent. In case of natural language it is hard to say, what the individuals are trying. And without explicit game description it is impossible to say how good somebody is in the game.

A research project which goes into the direction of game rule learning from natural language is “The Restaurant Game”. Here is the task to learn the rules of an unknown game only by observing the spoken language. The amount of papers is limited, but with new deeplearning methods and large datasets some progress seems to be possible.

Orkin, Jeff, and Deb Roy. "The restaurant game: Learning social behavior and language from thousands of players online." Journal of Game Development 3.1 (2007): 39-60.

$\endgroup$
0
$\begingroup$

Measuring and classifying human abilities related to intelligence have been done through a number of metrics.

  • Grades — When a student has very high grades, other students, teachers, and siblings tend to think and say that they are smart, even if they spend much time studying.
  • College readiness test results — This is actually the most tuned methodology and the best model because it is multidimensional, including separations for at least language skills and mathematics. Recent trends are to further divide skills into (a) command of evidence, (b) words in context, (c) expression of ideas, (d) use of standard language conventions, (d) algebra, (e) problem solving and data analysis, (f) advanced math, and (g) essay.
  • Intelligence test results — The trend to quantify intelligence in a single integer has been common, largely based on the single word intelligence being used in common language and jargon.
  • Assets — When a wealthy person speaks, others tend to listen, whereas very few of those indoctrinated into the philosophy of globalized industrialization would stop to listen to the wisdom of a homeless person.

Deciding Whether Aggregate Intelligence is Appropriate for the Use Case

One of the progenitors of aggregating the scores of varying test question types into a single number is Mensa. One can note this aggregating in their criteria for membership.

  • ACT Composite ≥ 29 prior to 9/1989
  • SAT Composite ≥ 1250 from 9/30/1974 to 1/31/1994
  • Miller Analogies Test ≥ 95% aggregated percentile after 10/2004

Test designers, aware that one may excel in one area of testing and not in another. The tests may randomize these question categories in the sequence they are presented or cluster them into sections, but all tests must cover a variety of question types to be useful in determining a person's alignment with a curriculum or role in the economy.

Consider that someone may qualify for Mensa based on their aggregate but may not if the same person doubled doubled one of the two major category scores, evidence based language or mathematics.

At age sixteen, my math abilities were so high, I qualify for Mensa solely because of its affect on the SAT aggregate and the award I received from the Mathematics Association of America. Consequently, the University of Connecticut accepted me for a dual engineering degree, but probably would not have for English Literature. Since then, I've become an avid reader and author. The variance in language and mathematics scores are not drastically skewed today.

In the use case described in the question, the appropriate question in response, as is often the case, is another question.

What exactly is the intended use of the extracted of low, medium, and high intelligence information?

The answer to that question will help determine whether to aggregate the various sections of intelligence tests or focus on a subset or single one. For instance, if the goal is to find bloggers or debt collectors, social leadership abilities are more important than evidence based language skills or mathematics. If the goal is to find programmers for a project, then low interest in socializing combined with high math and analytical skills is the best choice.

Upper management selection requires distinction between far reaching analytical skills and immediately applicable analytical skills. Nicola Tesla's electromagnetic designs, visible as 111 U.S. patents, power our lights and appliances worldwide, yet he ended in debt.

There are many factors that broaden the statistical variance in aggregate scoring.

  • Change in motivational environment
  • A mentally debilitating medical condition or finding a working management strategy for a previously unmanageable one
  • Drastic change in approach to life resulting from life-changing events

There are many factors that broaden the statistical variance between groups of tests too. There are at least 22 independent genetic controls that correlate significantly with intelligence scoring1, meaning that there are at least 4,194,304 possible genetic states that impact neural activity in the human brain's networks and chemistry correlated with test scores. They are not aligned along a single axis, but rather 22 axes with only two possible states per axis, with one of each state occurring dominantly.

Millions of Chat Message

In the case of chat, there are other variance sources that should be considered.

  • People with high linguistic skills in conversation know how to obscure their intelligence attributes to avoid triggering jealousy or aversion in the other chatters.
  • In chats with a higher requirement for wit, chatters can copy and paste or paraphrase text from other browser tabs to show intelligence characteristics they do not possess.
  • Since it is not a testing scenario from the chatter's perspective, habits like getting inebriated and then logging in to chat or those that chat to find online mates would skew the correlation between semantics and intelligence.

But these are not the largest challenges imposed by this question. Although there are methods in use for mapping series of words to semantic maps, the ability to correlate those maps with cognitive models and integrate such models with conversational ability is still in its infancy. A study of chatbot output is clearly indicative of the shortcomings. Here is a dialog from one of the most popular automated assistants.

Tell me a joke.

I don't really know any good jokes.

Oh wow you're pretty funny anyway.

Was it something I said?

Do you talk dirty?

The carpet needs vacuuming.

From the perspective of Alan Turing's thought experiment he called, "The imitation game," it is clear which is the person and which is the artificial device. The first reply indicated the appearance of comprehension of the verb, the noun, and the intention behind the sentence, however the intention was not recognized. The artificial device simply over-fit sufficiently to exhibit the appearance of intelligence. The lack of comprehension degrades from there.

Note that the human isn't exhibiting qualities that would qualify her or him for Mensa either, but the human is clearly leading the conversation and has an independent set of intentions that predate the conversation.

  • Finding amusement
  • Engaging sexually

One could not realistically determine the IQ of the human speaker from this particular chat through any known means.

Different Discussions

The variety in discussions may improve the likelihood of finding a reliable solution, however the differences in discussion sources must be kept with the dialogs as labels for that advantage to be useful.

This points to the second largest challenge in this project: Acquiring or constructing labelled data.

Already Labelled Data Sets

Although there are many data sets that are, "manually labelled multi-turn dialogue," the formal name for what the question seeks to avoid the manual labor, none were found with features labeled that strongly correlate to intelligence test results.

Some of the messages are written by people who lack understanding or relevant language skills. These messages almost always contain little to no thought and are almost always irrelevant to the main topic.

The human resource requirement to label millions of messages is large, and such labeling of intelligence is necessarily subjective. Stack Exchange Q&A fit within sites or sub-sites that outline a fairly narrowly defined field of interest, yet posts are voted both up and down regularly.

There may be a correlation between education level and either lexical structure or depth of vocabulary, but the subjective impression of that correlation is an untested hypothesis unless the chatters have accurately reported their testing scores from some standardized testing body.

Assumptions to Get Around These Challenges

If the project leads and stakeholders can accept the following two possibilities, the work can move forward with a few marginally ethical assumptions made.

  • Some percentage of those with high intelligence, who regularly express ideas to young children, the mentally challenged, or those adults who, due to cultural or economic realities, have limited vocabulary and cognitive ability may be marked as low intelligence.
  • Some percentage of those with low intelligence, who regularly interact with academics or family members with high education levels and may have learned to mimic their language patterns in limited ways may fool the learning system and be marked as high intelligence.
  • Some percentage of those with high intelligence may be tired, drunk, or just bored and without any motivation to impress the others in the conversation may ramble and therefore be marked as low intelligence.
  • Some percentage of those with low intelligence may copy intelligent writing, effectively paraphrase such, or recall memorized sentences from such and be marked as high intelligence.

Keep in mind that existing learners would have trouble achieving an IQ of five on any of the standardized tests without being trained specifically to score well on them. They do not understand. They react well when they have been trained on good data and have converged on a verifiably effective mapping surface that relates given input and expected output.

Assumptions to Make Progress

If the above false positives and false negatives are acceptable and the following assumptions are made, there may yet be progress toward the goal.

  • There is a reasonably reliable correlation between lexical constructs and the type of intelligence information of interest.
  • There is a reasonably reliable correlation between the extent of vocabulary and the type of intelligence information of interest.
  • A sufficient body of people, trained based on a specific criterion to be applied in the determination of intelligence of the type of interest, can be assembled to rate messages in the dialog.

Note that the trainees must rate each message in context and the training must be aware of previous dialog that the message author is likely to have read to train properly on the labels the trainees assign.

Algorithms

The most refined and mature algorithms for this type of historically aware learning is the LSTM class of algorithms.

Training the Trainees

Requirements and prerequisite for training in list form may be a helpful summary of the above.

  • Determine if an aggregation intelligence value or a particular intelligence related ability is of interest to the project leaders and stakeholders.
  • Develop a training program that causes ratings to follow a particular philosophy about estimating intelligence of the type of interest from the posts.
  • Clarification that context must be considered, such that each trainee is aware that they are expected to read until the dialog is comprehended before assigning a number.
  • Use of a scale of 1 to 5 instead of 1 to 3, which may improve training speed and accuracy.
  • Avoid use of terms like stupid and smart, which have semantic associations beyond basic intelligence.

For instance, a person may say, "Go take a long walk off a short pier." Another person may reply, "No, you." Although the reply may be perceived by some people as stupid, it may, by those same people be perceived as intelligent by those same people. The smart-stupid scale may not match the high-low intelligence scale. These two scales have some differences in the finer shades of semantics in common use of those words.

The dumb-average-intelligent categorization would be a mixing of linguistic axes. The clearest scales to use are like one of these.

  • Low intelligence
  • Medium intelligence
  • High intelligence

Or this one, which would produce a higher degree of input accuracy for training.

  • Very low intelligence
  • Somewhat low intelligence
  • Medium intelligence
  • Somewhat high intelligence
  • Very high intelligence

Further Research Recommendations

The question indicates an attachment to the idea of aggregated intelligence rating, which is not the most scientific or evidence-based formulation. The trend of testing is away from aggregation, even though qualifications for clubs and societies continue to require aggregate intelligence quantification to set a practical single bar for qualification.

The exact measuring of people's Intelligent Quotient by the way they write is a bit more interesting topic but I expect this would require far more research and fiddling.

It is correct that predicting IQ from chat is, "A bit more interesting," but may not be particularly useful in terms of selecting friends, employees, or participants in some event or intelligence experiment. What would be much more interesting and considerably more evidence-based would be predicting genetic states of the 22 or more genes based on chat, which would require three things, all three of which are also interesting.

  • Somehow partially labeling chat messages and their associated previous dialog with the DNA states of the determining genes and possibly their exposure to various nurture related elements like whether they had been enrolled in preschool, tutoring, patterns of Internet use, household income, family size, education level of the teachers in the schools attended, and other factors that could show potentially useful correlations not yet found.
  • Development of a way of relating semantic maps between messages in a sequence and finding a relationship between the message's semantic map relationships and the presence of external intelligence indicators in the message authors.

References

Genome-wide association meta-analysis of 78,308 individuals identifies new loci and genes influencing human intelligence, Suzanne Sniekers et. al., 2017

$\endgroup$
  • 1
    $\begingroup$ The SAT score doesn't measure intelligence, it measures the ability of schools to educate people. That means, the university has a capacity of 1000 free places, but 10000 applications, and the SAT score is used to tell 9000 of them, that they are not welcome. Additionally, the answer can be shorten to 10% which would increase the comprehensibility. $\endgroup$ – Manuel Rodriguez Nov 1 '18 at 14:46
  • 2
    $\begingroup$ The one who downvoted this awesomely elaborate answer - could you explain why ? $\endgroup$ – Boris D. Teoharov Nov 1 '18 at 15:27
  • 1
    $\begingroup$ @DouglasDaseeco The answer elaborates too much about human societies but too little about theoretical foundation of language games. A signaling game is something which can be played by humans, by bad playing chatbots or by highly developed AI, see "Stergos, et al.: Developing a corpus of strategic conversation in the settlers of catan, 2012" There is no need to discuss in the answer the higher education system. If i could, i would downvote it a second time. $\endgroup$ – Manuel Rodriguez Nov 1 '18 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.