This is an important question for AI – maybe the most important of all – for the research field of Artificial Intelligence. I mean if AI is science, then its experiments will be empirically testable. There has to be a way to decide pass or fail. So what are the tests for intelligence? Before you even design a test, you need a clear idea of what intelligence amounts to, otherwise how could you design a competent test for it?
Sure, I'm part of the research and development project known as Building Watertight Submarines, and sure, I'm totally confident my submarine is watertight, but I have no idea how to test whether it is or not because I don't know what "watertight" means.
This whole idea is absurd. But ask AI what "intelligence" means. The answers you get, on analysis, are almost the same as the submarine example.
Base Answer - Behavior
The word (idea, concept) "Intelligence" is usually defined by AI in terms of behavior. I.e. the Turing test approach. A machine is intelligent if it behaves in a way that, were a human to behave in that same way, the human would be said to be performing an action that required human intelligence.
Problem 1: player pianos are intelligent. Playing a Scott Joplin tune obviously requires intelligence in a human.
Problem 2. If a machine passes the test, it only shows that the machine is "intelligent" for the tested behaviors. What about untested behaviors? This is actually a life-and-death problem today with self-driving vehicle AI control systems. The AI systems are acceptably good at driving a car (which obviously requires human intelligence) in specific environments, e.g. freeways with well-marked lanes, no tight corners, and a median barrier separating the two directions. But the systems go disastrously wrong in "edge cases" – unusual situations.
Problem 3. Who would put their child on a school bus driven by a robot that had passed the Turing test for driving school buses? What about a storm when a live power line falls across the road? Or a twister in the distance is coming this way? What about a thousand other untested possibilities? A responsible parent would want to know (a) what are the principles of the internal processes and structures of human intelligence, and (b) that the digital bus driver had adequately similar internal processes and structures – i.e., not behavior but the right inner elements, the right inner causation.
Desired answer – inner principles
I would want to know that the machine was running the right inner processes and that it was running these processes (algorithms) on the right inner (memory) structures. Problem is, no one seems to know what the right inner processes and structures of human intelligence are. (A huge problem to be sure – but one that hasn't held AI back – or self-driving system developers - one bit.) The implication of this is that what AI ought to be doing now is working out what are the inner processes and structures of human intelligence. But it's not doing this – rather, it's commercializing its flawed technology.
Elements of a definition – 1. Generalization
We do know some things about human intelligence. Some tests really do test whether a machine has certain properties of the human mind. One of these properties is generalization. In his 1950 paper, Turing, as a sort of joke, gave a really good example of conversational generalization: (The witness is the machine.)
"Interrogator: In the first line of your sonnet which reads ' Shall I compare thee to a summer's day ', would not ' a spring day ' do as well or better?
Witness : It wouldn't scan.
Interrogator : How about ' a winter's day ' That would scan all right.
Witness: Yes, but nobody wants to be compared to a winter's day.
Interrogator: Would you say Mr. Pickwick reminded you of Christmas?
Witness: In a way.
Interrogator: Yet Christmas is a winter's day, and I do not think Mr. Pickwick would mind the comparison.
Witness : I don't think you're serious. By a winter's flay one means a typical winter's day, rather than a special one like Christmas."
Current AI has nothing that comes even remotely near being able to generalize like this. Failure to generalize is regarded as perhaps the greatest failing of current AI. The ability to generalize would be one part of an adequate definition of "intelligence". But what generalization amounts to would need to be explicated.
The problem of generalization, also, is behind several the severe philosophical objections to AI theory, including the frame problem, the problem of common-sense knowledge, and the problem of combinatorial explosion.
Elements of a definition – 2. Perception
Sensory perception is fairly obviously fundamental to human learning and intelligence. Data (in some form) is emitted by the human senses then processed by the central system. In the computer, binary values exit the digital sensor and travel to the machine. However, nothing in the values themselves indicates what was sensed. Yet the only thing the computer gets is the binary values. How could the machine ever come to know what is sensed? (The classic Chinese room argument problem.)
So another element of human-like intelligence is the ability to perceive in a human-like way. What "human-like way" means here is that the machine processes sensory input using the same principles that apply in human perception. The problem is that no one seems to know how a semantics (knowledge) can be built from the data emitted by digital sensors (or organic senses). But still, human-like perception needs to be an element of an adequate definition of "intelligence".
Once AI gets these two issues sorted out – generalization and perception – then it will probably, hopefully, be well on the way to realizing its original goal of almost 70 years past – building a machine with (or that could acquire) a human-like general intelligence. And maybe the principles of generalization and the principles of perception are one and the same. And maybe there is actually only one principle. It shouldn't be assumed that the answers are complex. Sometimes the hardest things to understand are the most simple.
So the question "What do we mean when we say "intelligence"? is really important to AI. And the conclusion is that AI ought to replace its current behavioral definition of "intelligence" with one that includes the human elements of generalization and perception. And then get on and try to work out the operating principles, or principle, of both of these.