138

This classic problem exhibits a basic misunderstanding of what an artificial general intelligence would likely entail. First, consider this programmer's joke: The programmer's wife couldn't take it anymore. Every discussion with her husband turned into an argument over semantics, picking over every piece of trivial detail. One day she sent him to the ...


68

How could self-driving cars make ethical decisions about who to kill? It shouldn't. Self-driving cars are not moral agents. Cars fail in predictable ways. Horses fail in predictable ways. the car is heading toward a crowd of 10 people crossing the road, so it cannot stop in time, but it can avoid killing 10 people by hitting the wall (killing the ...


52

The answer to a lot of those questions depends on how the device is programmed. A computer capable of driving around and recognizing where the road goes is likely to have the ability to visually distinguish a human from an animal, whether that be based on outline, image, or size. With sufficiently sharp image recognition, it might be able to count the number ...


49

This popular meme originated in the era of 'Good Old Fashioned AI' (GOFAI), when the belief was that intelligence could usefully be defined entirely in terms of logic. The meme seems to rely on the AI parsing commands using a theorem prover, the idea presumably being that it's driven into some kind of infinite loop by trying to prove an unprovable or ...


29

Personally, I think this might be an overhyped issue. Trolley problems only occur when the situation is optimized to prevent "3rd options". A car has brakes, does it not? "But what if the brakes don't work?" Well, then the car is not allowed to drive at all. Even in regular traffic, human operators are taught that your speed should be limited as such that ...


25

I see several good answers, but most are assuming that inferential infinite loop is a thing of the past, only related to logical AI (the famous GOFAI). But it's not. An infinite loop can happen in any program, whether it's adaptive or not. And as @SQLServerSteve pointed out, humans can also get stuck in obsessions and paradoxes. Modern approaches are ...


18

The halting problem says that it's not possible to determine whether any given algorithm will halt. Therefore, while a machine could conceivably recognize some "traps", it couldn't test arbitrary execution plans and return EWOULDHANG for non-halting ones. The easiest solution to avoid hanging would be a timeout. For example, the AI controller process could ...


15

In the real world, decisions will be made based on the law, and as noted over on Law.SE, the law generally favors inaction over action.


15

This is the well known Trolley Problem. As Ben N said, people disagree on the right course of action for trolley problem scenarios, but it should be noted that with self-driving cars, reliability is so high that these scenarios are really unlikely. So, not much effort will be put into the problems you are describing, at least in the short term.


14

Another similar question might be: "What vulnerabilities does an AI have?" "Kill" may not make as much sense with respect to an AI. What we really want to know is, relative to some goal, in what ways can that goal be subverted? Can a paradox subvert an agent's logic? What is a paradox, other than some expression that subverts some kind of expected behavior?...


10

Nope in the same way a circular reference on a spreadsheet cannot kill a computer. All loops cyclic dependencies, can be detected (you can always check if a finite turing machine enters the same state twice). Even stronger assumption, if the machine is based on machine learning (where it is trained to recognize patterns), any sentence it is just a pattern ...


9

For a driverless car that is designed by a single entity, the best way for it to make decisions about whom to kill is by estimating and minimizing the probable liability. It doesn't need to absolutely correctly identify all the potential victims in the area to have a defense for its decision, only to identify them as well as a human could be expected to. ...


9

“This moral question of whom to save: 99 percent of our engineering work is to prevent these situations from happening at all.” —Christoph von Hugo, Mercedes-Benz This quote is from an article titled Self-Driving Mercedes-Benzes Will Prioritize Occupant Safety over Pedestrians published OCTOBER 7, 2016 BY MICHAEL TAYLOR, retrieved 08 Nov 2016. Here's an ...


9

Google’s self-driving car most likely uses mapping of traffic signs using google street view images for roadway inventory management. If traffic signs are not in its database, it can still “see” and detect moving objects which can be distinguished from the presence of certain stationary objects, like traffic lights. So its software can classify objects based ...


9

No. This is easily prevented by a number of safety mechanisms that are sure to be present in a well-designed AI system. For example, a timeout could be used. If the AI system is not able to handle a statement or a command after a certain amount of time, the AI could ignore the statement and move on. If a paradox ever does cause an AI to freeze, it's more ...


8

AIs used in computer games already encounter similar problems, and if well designed, they can avoid it easily. The simplest method to avoid freezing in case of an unsolvable problem is to have a timer interrupt the calculation if it runs too long. Usually encountered in strategy games, and more specifically in turn based tactics, if a specific move the ...


6

How could self-driving cars make ethical decisions about who to kill? By managing legal liability and consumer safety. A car that offers the consumer safety is going to be a car that is bought by said consumers. Companies do not want to be liable for killing their customers nor do they want to sell a product that gets the user in legal predicaments. Legal ...


5

Frankly I think this issue (the Trolley Problem) is inherently overcomplicated, since the real world solution is likely to be pretty straightforward. Like a human driver, an AI driver will be programmed to act at all times in a generically ethical way, always choosing the course of action that does no harm, or the least harm possible. If an AI driver ...


5

They shouldn't. People should. People cannot put the responsibilities of ethical decisions into the hands of computers. It is our responsibility as computer scientists/AI experts to program decisions for computers to make. Will human casualties still exist from this? Of course, they will--- people are not perfect and neither are programs. There is an ...


5

It seems to me this is just a probabilistic equation like any other. I'm sure Google handles paradoxical solution sets Billions of times a day, and I can't say my spam filter has ever caused a (ahem) stack overflow. Perhaps one day our programming model will break in a way we can't understand and then all bets are off. But I do take exception to the ...


5

Well, the issue of anthropomorphizing the AI aside, the answer is "yes, sort of." Depending on how the AI is implemented, it's reasonable to say it could get "stuck" trying to resolve a paradox, or decide an undecidable problem. And that's the core issue - decidability. A computer can chew on an undecidable program forever (in principle) without ...


4

Killing AI by 'thinking' about a paradox would be called a bug in implementation of that AI, so it's possible (depending how it's being done), but less likely. Most of AI implementation operate in non-linear code, therefore there is no such thing as an infinite loop which can "freeze" the computer's 'consciousness', unless code managing such AI consist ...


4

I think that in most cases the car would default to reducing speed as a main option, rather than steering toward or away from a specific choice. As others have mentioned, having settings related to ethics is just a bad idea. What happens if two cars that are programmed with opposite ethical settings and are about to collide? The cars could potentially have a ...


3

I think there would not be a way to edit such ethics settings in a car. But hey, if cell phones can be rooted, why not cars? I imagine there'll be Linux builds in the future for specific models that will let you do whatever you want. As for who'll make such decisions, it'll be much like privacy issues of today. There'll be a tug-of war on the blanket by the ...


3

The only sensible choice is to use predictable behaviour. So in the people in front of the car scenario: First the car hits the brakes, at the same time honks the horn, and stays on course. The people then have a chance to jump out of the way leading to zero people being killed. Also with full brakes (going from 50km per hour to zero is less than 3 car ...


3

Suppose you have data: color height quality ===== ====== ======= green tall good green short bad blue tall bad blue short medium red tall medium red short medium To calculate the entropy for quality in this example: X = {good, medium, bad} x1 = {good}, x2 = {bad}, x3 = {medium} Probability of each x in X: p1 = 1/6 = 0....


3

This question is re-inventing the analysis for iterated prisoner's dilemma and the co-evolution that can lead to agents playing super-rationally in the one-shot version, which has been studied really extensively. Dan Ashlock's research career looked at this in great detail from an evolutionary perspective, but it's also been widely studied in other areas ...


2

It depends on whether the loss of the robot would end up causing harm to humans. If the robot was supposed to be watching for a suspected terrorist attack to start taking place (so it could alert authorities or halt the attack), it would be very bad if somebody dismantled the robot or otherwise stopped it from carrying out its mission. In that case, the ...


2

The question mentions "walking robot", but it may be illustrative to re-frame the discussion in terms of self-driving cars, because: It gives a common point of reference, rather than everyone having their own separate vision of how vulnerable/powerful a kung-fu walking robot might be. We already know a lot about societal attitudes to car theft. Given that ...


Only top voted, non community-wiki answers of a minimum length are eligible