Why was ELIZA able to induce "delusional thinking"?

Question

From Eliza to A.L.I.C.E.:

Weizenbaum tells us that he was shocked by the experience of releasing ELIZA (also known as "Doctor") to the nontechnical staff at the MIT AI Lab. Secretaries and nontechnical administrative staff thought the machine was a "real" therapist, and spent hours revealing their personal problems to the program. When Weizenbaum informed his secretary that he, of course, had access to the logs of all the conversations, she reacted with outrage at this invasion of her privacy. Weizenbaum was shocked by this and similar incidents to find that such a simple program could so easily deceive a naive user into revealing personal information.

Wikipedia's article on the "ELIZA Effect":

Though designed strictly as a mechanism to support "natural language conversation" with a computer, ELIZA's DOCTOR script was found to be surprisingly successful in eliciting emotional responses from users who, in the course of interacting with the program, began to ascribe understanding and motivation to the program's output. As Weizenbaum later wrote, "I had not realized ... that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people." Indeed, ELIZA's code had not been designed to evoke this reaction in the first place. Upon observation, researchers discovered users unconsciously assuming ELIZA's questions implied interest and emotional involvement in the topics discussed, even when they consciously knew that ELIZA did not simulate emotion.

ELIZA, despite its simplicity, was incredibly successful at its task of tricking other human beings. Even those who knew ELIZA was a bot would still talk to it. Obviously, ELIZA served as an inspiration for various other, more intelligent chatbots, such as Xiaoice. But I would like to know what exactly led to such a simple program like ELIZA to be so successful in the first place.

This is very useful knowledge for a programmer since a simple program is one that would be easily maintainable.

Answer 1

I like your choice of "induce" instead of "produce," because the delusions came from the users. This means the answer has to do mostly with human psychology; people come equipped with lots of mental machinery specialized for dealing with other humans and not very much mental machinery specialized for dealing with software. So ELIZA behaved in ways that some people classified it as a person and behaved accordingly, and others didn't.

What features will trip up a person's internal person classification system seem like they vary heavily from person to person, and also with experience and familiarity. Going into more detail is, as mentioned in the comments, more appropriate for sites specializing on the human side of the keyboard.