If we were to build an AGI with a reward function that's "aligned" with human values, are there any not-extremely-unlikely routes to hyperexistential catastrophe stemming from some sort of sign error-type bug? Things like accidentally putting the loss function where the reward is expected, or flipping a boolean flag's meaning in a database without updating downstream callers (for an AGI using online learning), etc.

How common are these errors with current AI systems, and would you expect them to plausibly occur with an extremely important AI system after training & deployment? What I'm imagining isn't human extinction, but more "we create a friendly AI but screw up and instead cause it to do the worst possible thing."

  • $\begingroup$ Are you aware of the paperclip thought experiment? Also, it's not clear to me if you're asking for "human error" while implementing the AGI or if you're asking if an AGI could make a mistake while manipulating its source code, assuming it can manipulate its source code. Please, clarify these issues. $\endgroup$ – nbro Aug 18 at 0:09
  • $\begingroup$ @nbro I'll elaborate: yes, I'm aware of the paperclip maximiser concept. The AI I'm imagining is one that actively minimises human values as a result of a SignFlip-type error. I'm imagining an AI with a utility function almost aligned with human values but, due to human or AI error, optimises the negative of human values a la "I have no mouth and I must scream". I'm asking about all possible causes of this sort of error, which would include both human error in the implementation and errors in the AI manipulating its source code. $\endgroup$ – Anirandis Aug 18 at 1:07
  • $\begingroup$ So, are you asking for all types of programming and logic errors? I think that's a little too broad. And why are you asking about the "inverse of its reward function" in the title? Maybe saying "inverse" is a bit too strict, given that we are probably no close to building an AGI. $\endgroup$ – nbro Aug 18 at 10:16
  • $\begingroup$ @nbro Sure, it's somewhat speculative. It's not asking about all types of errors, but rather whether there are any errors that would be likely to cause an advanced machine learning software to implement the opposite of its reward function after the system has been trained and deployed. $\endgroup$ – Anirandis Aug 18 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.