It seems fairly uncontroversial to say that NN based approaches are becoming quite powerful tools in many AI areas - whether recognising and decomposing images (faces at a border, street scenes in automobiles, decision making in uncertain/complex situations or with partial data). Almost inevitably, some of those uses will develop into situations where NN-based AI takes on part or all of the human burden, and generally does it better than people generally do.
Examples might include NN hypothetically used as steps in self-driving cars, medical diagnosis, human/identity verification, circuit/design verification, dubious transaction alerting. Probably many fields in the next decade or so.
Suppose this happens, and is generally seen as successful (for example, it gets diagnoses right 80% to human doctors' 65% or something, or cars with AI that includes an NN component crash 8% less than human-driven cars or alternatives, or whatever).
Now - suppose one of these aberrantly and seriously does something very wrong in one case. How can one approach it? With formal logic steps one can trace a formal decision process, but with NN there may be no formal logic, especially if it gets complex enough (in a couple of decades say), there are just 20 billion neural processors and their I/O weightings and connections, it may not be possible to determine what caused some incident even if lives were lost. It also may not be possible to say more than the systems continually learn and such incidents are rare.
I also haven't heard of any meaningful way to do a "black box" or flight recorder equivalent for NNs, (even if not used I a life-critical case), that would allow us to understand and avoid a bad decision. Unlike other responses to product defects, if a NN could be trained after the event to fix one such case, it doesn't clearly provide the certainty we would want, that the new NN setup has fixed the problem, or hasn't reduced the risk and balance of other problems in so doing. It's just very opaque. And yet, clearly, it is mostly very valuable as an AI approach.
In 20 years if NN is an (acknowledged as safe and successful) component in a plane flight or aircraft design, or built into a hospital system to watch for emergencies, or to spot fraud at a bank, and has as usual passed whatever regulatory and market requirements might exist and performed with a good record for years in the general marketplace, and then in one case such a system some time later plainly misacts on one occasion - it dangerously misreads the road, recommends life-damaging medications or blatantly misdiagnoses, or clears a blatant £200m fraudulent transaction at a clearing bank that's only caught by chance before the money is sent.
What can the manufacturer do to address public or market concerns, or to explain the incident? What do the tech team do when told by the board "how did this happen and make damn sure it's fixed"? What kind of meaningful logs can be kept, etc.? Would society have to just accept that uncertainty and occasional wacky behavior could be inherent (good luck with the convincing society of that!)? Or is there some better way to approach logging/debugging/decision activity more suited to NNs?
