Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
Do we know why Tesla's autopilot mistaken empty sky with a high-sided lorry which resulted in fatal crash involving a car in self-drive mode? Was it AI fault or something else? Is there any technical explanation behind this why this happened?
The articles Tesla Driver In First Self-Drive Fatal Crash and Tesla driver killed in crash with Autopilot active, NHTSA investigating talk about this topic, but do not go in detail.
Tesla's technology is assistive, as Alexey points out, so this is not a case of an autonomous system (e.g. an AGI) doing some fatal stunt (the product name AutoPilot is famously misleading). Now on why the car assistance led to this tragic accident, there is some information related to AI technologies.
Warning: I cannot find again the source critical to the next paragraph, and reading again pages over pages, I cannot find similar argument in other reports. I still remember vividly the point below, but please keep in mind it may be incorrect. The rest of the answer is weakly related, so I leave it all, with this warning.
An independent report (link needed, I can't find it...) explained that the assistive system was unable to detect the truck due to an exceptionally low contrast (bright sky perceived as white---colour of the truck). The report also said that a human driver would have been unable to make the difference either. In other words, it is possible that car sensors (presumably camera) and the human eye could not have detected an obstacle, and could not have triggered any safety measure. This short graphical explanation sums up the car sensors: Camera, radar, GPS, etc.
The assistive sub-system is based on proprietary AI technologies. We can only speculate under some hypothesis. _This is not very useful, honestly, except for illustration purpose. Assuming the assistive system relies on ML technologies to learn about obstacles from a video stream (such systems do exist):
Why those technologies did not work in that case will remain a secret. We can say however that any system---whether built with AI technology or not---has limits. Beyond these limits, the system reaction is unpredictable: It could stop, reset, shutdown. The difficulty here is to define what a "default behaviour" is. A machine will basically do whatever it is designed to do, so an AI-based system too.
We could speculate even more on what would happen if the assistive system was really autonomous, the elusive AGI, but that is really not the case here.
As far as I know, Tesla cars autopilot is not a 100% AI pilot, it's an assitant: as it detects hands off wheel it slows down, so it's incorrect to speak about AI mistake: it is not trained/designed to drive a car all by itself. A human driver is responsible in that incident.
Tesla model S has Autopilot which allows to steer within a lane, change lanes with the simple tap of a turn signal, and can manage speed by using traffic-aware cruise control. Multiple digital controls helps to avoid collisions. Based on that, this isn't fully self-driving car.
However it is using a computer vision detection system, but it is not intended to be used hands-free.
So basically what is known is that the accident involved the side of a truck trailer (of a large white 18-wheel truck) and most likely the camera had a washed out of picture possibly due to glare or blooming from overexposure which made that the side of the trailer white and thin which failed to distinguish with the sky which was bright as well.
This may have happened in part, because the crash-avoidance system only engage when both radar and vision system detect an obstacle which could not happen.
Further more it was suggested by The Associated Press that the driver most likely was watching a Harry Potter at the time of the crash and assuming system would alert Brown, we don't know if he was able to retake controls quickly enough to avoid impact. As mentioned again, the system wasn't intended for hands-free driving and parts of the system was unfinished. Not to mention that the car was driving with full speed under the trailer.
Tesla officially said about this crash in a statement on its website:
The high ride height of the trailer combined with its positioning across the road and the extremely rare circumstances of the impact caused the Model S to pass under the trailer, with the bottom of the trailer impacting the windshield of the Model S.
Neither Autopilot nor the driver noticed the white side of the tractor-trailer against a brightly lit sky, so the brake was not applied.
They also said, according to techno-optimists, that they will tweaks their code, so this particular case won't happen again.
To summarize, this was a 'technical failure' of braking system and most likely Autopilot was not at as Tesla told Senate.
The New York Times |Source: Florida traffic crash report
Sources:
