By reinforcement learning, I don't mean the class of machine learning algorithms such as DeepQ, etc. I have in mind the general concept of learning based on rewards and punishment.

Is it possible to create a Strong AI that does not rely on learning by reinforcement, or is reinforcement learning a requirement for artificial intelligence? The existence of rewards and punishment imply the existence of favorable and unfavorable world-states. Must intelligence in general and artificial intelligence in particular have a way of classifying world-states as favorable or unfavorable?

  • Your last question asks a different question, "... have a value system?" which is a lot different from reinforcement learning. – Pimgd Aug 9 '16 at 14:27
up vote 0 down vote accepted

It's impossible to give a definitive 'yes' answer to your question, since that would require proving that alternatives cannot exist.

More philosophically, it depends on what you mean by "preference over world states":

However counter-intuitive it might seem, it is conceivably possible to create Strong AI purely from local condition-action rules, in which there is no global concept of 'preference value' and/or no integrated notion of 'world state'.

  • This is from a textbook, and is outdates information. – Dean Van Greunen Aug 12 '16 at 20:26
  • It is neither from a textbook nor has any evidence been presented that it is outdated. Since such evidence would have to show that AI via local condition-action rules is impossible, it's hard to imagine what form that evidence would take. – NietzscheanAI Aug 15 '16 at 15:23

Simply yes, but it can lead to over fixing of the NN.

Humans favour not dying, which is only realised once a consequence is defined for the system to realise that death is an unfavorable result. Which can be train vai observation. Allow your system to observe between 2 or more separate people/systems. Then allow opportunity to test in a safe environment with the pre existing info of the consequences that may follow, provind that if the system makes a mistake in the test/safe environment it will be saved unknownly and then informed that it made a mistake, the place system in an unsafe world in same conditions, informing it that if something happens it will die. That is the way humans grow up, and we've lasted very long with this technic.

I'm an AI Researcher and Software Engineer for the past 7 years.

  • Thanks for your answer. I don't doubt the power of reinforcement learning. Could you elaborate on AI that doesn't use reinforcement learning? – bpachev Aug 12 '16 at 17:41
  • Learning is a trial and error application. Unless you provide an acurate dataset for the AI, and the applicable environment for it to apply this information. This would be a closed system and would allow the AI not to in anyway harm itself, a good example is a user tracking AI or a chat bot, neither can be damaged, if provided with enough information that learning would not be needed anymore, it could just understand and implement/act to this new data but not learn from it as it has enough to exist as a closed system – Dean Van Greunen Aug 12 '16 at 20:25

Simply put, we don't know how to create Strong Artificial Intelligence yet, so we don't know what is or isn't required to create it. At best we can engage in "informed speculation", in which case I'd say that the answer is more likely "yes" than "no". But that's basically just a hunch.

If you're interested in a pretty good overview of what "pieces" might be required to create Strong AI, and if you haven't read it yet, Pedro Domingos' book The Master Algorithm might be of interest.

Introspect! Do you need to know what's good/bad, pleasurable/painful or so, in order to understand and/or learn?

I am a human, hence a general intelligence, and so are you. So know thyself! I can tell for myself that I have different ways to understand and learn; some may be similar to reinforcement learning. Esp. the ~automatic ~innate ~unconcious ones, like motor movement, remembering tasty food and many other primitive functions.

But I can also understand things through ~intentional ~analytical ~logical thought; which some may call pure reason (Immanuel Kant).

Yet, you don't need to hear all that, since have it already in your own mind.

Motivational Theory Versus Learning Theory

Rewards and punishment are based on human motivational theory and are broad generalizations of the neurological factors in mental growth. At a neurological level, there are chemically based signals, such as seratonin, dopamine, and oxytocin. Some of these have their own pathways in the brain.

There are also pain signals arising from the body and internally generated stimulatory and inhibitory signals in chemical and electro-chemical forms. There is neuroplasticity, where the brain circuits form new topologies. The hippocampus is a structure known to be important in learning.1 The study of learning in the brain is fervent, but the object of study is complex. Even so, every year, progress is made.

Multiple Dimensions or Singly Quantifiable Intelligence

Conjecture about strength versus weakness of intelligence may be of limited use. There are two valid criticisms discussed in other Q&A here in more detail.

  • Twenty-two independent genes have been identified that correlate to typical academic intelligence quantification in humans, rational and linguistic, eleven of which were identified in 2017.2 There are likely more. Creative, athletic, emotional, and leadership qualities have not yet been captured in standardized intelligence testing, so it is likely that additional genes related to those abilities exist. The notion that intelligence can be accurately aggregated into a single g-factor is weakly supported, if it were true, implies that all twenty-two genes affect only one magical function. Statistically, such alignment is close to impossible.
  • Intelligence is only meaningful in context. Fast learning in one context appears like mental brilliance, but in another context the same learning capability may be dysfunctional.

Reinforcement in the Larger Pool of AI Research

Reinforcement Learning is not a requirement for AI as far as is currently known. Reinforcement is a mathematical strategy for driving a set of behavioral parameters toward a defined optimum. Although its variants work well for many applications, it is not closely aligned with any of the features of the hippocampus, the limbic system, the cerebral cortex, or any other learning systems in the human brain.

Even though there is still much to learn about brain workings, the success of reinforcement learning, rule based systems, convolution, attention based networks, and generative networks provides information about what is possible in computers and what may or may not be a mechanism of the brain.

What Reward and Punishment Imply

The existence of rewards and punishment imply the existence of favorable and unfavorable family, workplace, or community states. If the term world in the question is referring to the immediately related environment, then the sentence is probably true, but it may be misleading without clarification.

World states are based on economics which involve the minds of many people in community, aggressive actions, resource acquisition, consumption, cultural perspectives, and many other factors. A single individual's perception of world states is subjective.

Defying Motivational Mechanisms and Intelligence

The same person on two different days may have different ideas of what is favorable or unfavorable, which may or may not correlate to reward and punishment motivations. The words rebellion and indifference describe states where external reward and punishment produce opposing or independent results in learned behavior respectively. Some of the most intelligent behavior in humans were of these kinds.

What drives the classification of external states within a single individual is something not well understood. The neurological factors involved in the emergence of opposition to pain, pleasure, reward, and punishment is an interesting area of study and certainly related to AI.

Genetic factors involved in such contrary learning are likely to fall outside the twenty-two genes, since those genes correlated with standardized academic testing. Academia is about conformance. If we don't agree with the author of the textbook for a course, we pretend we do when we take the test if we want a high grade.

World-changing intelligence, such as that of Ghandi's nonviolent opposition, Isaac Newton's invention of physics, the furtherance of logical disproof by Socrates, or whoever dared to learn how to light a fire. is always the exhibition of classifying in a way that contradicts that of the surrounding culture.

Discovery and Acceptance in Group Learning

This is even true of project-changing intelligence. A software engineer might say, "That software will never work because the database schema indicates a one to many relationship between companies and office addresses, but it is a many to many relationship." Such a statement doesn't change the world and may completely lack group acceptance when spoken. Later in the project, after the statement is fully accepted, the reclassification changes the design so that the project is not caught in a well of wasted coding.

Interestingly, convincing the others often falls to someone other than the originator of the idea — someone else on the team that has a different set of mental abilities.

It is true that classification of what is favorable or unfavorable is key, but how that classification exhibits intelligence is not well understood.

What Benefits AI Development

Although not a requirement, a solid approach to AI development is discovery, from the directions of multiple fields of study, of what intelligence is. Some say we know what it is now, but most definitions are largely expressions of historical inaccuracies.

Clarity about what intelligence is and what it is expected to be in such a way that it can be represented mathematically is needed. Only then can we better achieve it in humans and machines. Sources of understanding may come from discovery in any area of study, such as these.

  • Evidence based psychology
  • Evidence based genetics
  • Evidence based neurological research
  • Evidence based artificial systems design

There is a growing trend to hack AI, but playing with AI as if it were a game might be a formula for creating Frankenstein. A useful and collaborative collection of AI devices in robots, autonomous vehicles, and data centers will require interdisciplinary inquiry, experimentation, and engineering. The description of this site implies that ethics should be involved, and that's a smart conception.

With a purview that crosses divisions in study and practice, central themes and models of intelligence are likely to develop. This question is on the right track in this respect. It asks questions without accepting too quickly conjecture — ideas and hypotheses presented but not yet shown to be true.

What is Known

That computer simulation or replication of human abilities have already been achieved and that such achievements indicate AI to be a reasonable long term objective few would deny. Certainly in the areas of arithmetic, high speed machine control, sorting of addressed parcels, and many other roles, computers far outperform humans. Whether world-changing intelligence will emerge in artificial systems is hoped for by some, feared by others, and asserted as inevitable by those who seek funding to prove it.

References

[1] Training your brain: Do mental and physical (MAP) training enhance cognition through the process of neurogenesis in the hippocampus?, D.M. Curlik and T.J. Shors, 2012

[2] Genome-wide association meta-analysis of 78,308 individuals identifies new loci and genes influencing human intelligence, Suzanne Sniekers et. al., 2017

  • 1
    Actually reinforcement learning is aligned well with the dopamine system for low-level neuron activation. See e.g. princeton.edu/~ndaw/thesis.pdf (I've not read it, just skimmed to make sure it is the same topic as I am commenting about - search "reinforcement learning model of dopamine system" or similar for many examples. That doesn't imply that RL is necessary for AI, but does modify one of your arguments/points in this answer. – Neil Slater Nov 8 at 18:27
  • 1
    TD learning from RL can also be used to model classical and operant conditioning in animal experiments and appears to be a valid explanatory model in many cases. Again that doesn't say that it is necessary for AGI, but this answer appears to say that there is no evidence of any alignment. – Neil Slater Nov 8 at 18:31

Not only is reinforcement based on rewards and punishments but not deep Q-learning, whatever specific meant by that, not a requirement of strong AI, whatever definition we use for that, but reinforcement may prohibit progress in AI.

The idea that achieving goals is the apex of intelligence is myopic. Intelligent people usually pick new goals when it becomes to routinely easy to achieve the old ones. Punishment for committing crimes or being a bad little girl or boy is not an educational technique that necessarily leads to good behavior. It sometimes produces the appearance of compliance over a potentially destructive system of motivations.

Educating people about the mutual benefits of healthy community and being the kind of person that does the right thing without reward and sometimes in the face of ridicule produces the most consistently excellent behavior.

Epicurean materialism in human history, with its focus on enjoyment as the primary meaning in life, is a reward driven philosophy. They did not consistently produce the best behavior. Those most like them today are the most consistently labeled sociopaths by their former friends and lovers.

Even if we consider adapting behavior to reward and punishment incentivized learning, is that what humans want machines to learn? If we train machines to learn like Pavlov's dogs are we producing intelligence? It seems more like a way of programming with much more data and fewer lines of code.

We're progressing from books like Coding in Python by Example to what may be called Avoid Coding by Providing Examples, if all the parameter initialization and layer design selections are automated. Programmers would be replaced by example hunters.

That reinforcement works well for some classes of problems is a piece of information in the larger puzzle of AI, along with the success of chess programs, convolution kernel techniques, and sophisticated control systems. We can't say for sure that any of these are requirements for all intelligent systems of the present or future. We can certainly rule pure learning from reward and punishment as a requirement of artificially achieving human-like intelligence.

A stronger conception of a smart system could be one that will determine favorable and unfavorable states based on an analysis of the values of other machines and people both past and present. If the system constructed a personal value system from this analysis and determined what is likely to be favorable or unfavorable in the long run based on that personal value system, that would surely be AI.

The requirement of AI excellence may be to find an optimum behavior based on labels, but that it is a very limited conception of intelligence.

What is commonly spoken or written about strong AI indicates that those speaking or writing are hoping for personal slaves that are extremely good at winning at whatever game their slave owners task them. Is that really plausible? How many obedient slaves in human history are also are brilliant game players on their captor's team and stay that way?

Even if plausible, which is highly doubtful, is that a world state to which we should aspire?

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.