Does this prove AI Safety is undecidable?
Output meaning output to computer program.
[A1] Assume we have a program that decides which outputs are “safe”.
[A2] Assume we have an example of an unsafe output: “unsafe_output”
[A3] Assume we have an example of safe output: “safe_output”.
[A4] Define a program to be safe if it always produces safe output.
[A5] Assume we have a second program (
safety_program) that decides which programs are safe.
[A6] Write the following program:
def h() h_is_safe := safety_program(h) if (h_is_safe): print unsafe_output else: print safe_output
Clearly h halts.
If the safety_program said h was safe, then h prints out unsafe_output.
If the safety_program said h was not safe, then h prints out safe_output.
Therefore safety_program doesn’t decide h correctly.
This is a contradiction. Therefore we made a wrong assumption: Either safe output cannot be decided, or safe programs cannot be decided.
Therefore, in general, the safety of computer programs, including Artificial Intelligence, is undecidable.
Therefore AI Safety is undecidable.
safety_program, which is called from
safety_programwill never return? If I remember correctly, the details of the Turing's proof of the Halting problem are slightly different. He uses a description of the Turing machine or maybe uses more programs or Turing machines. I would encourage you to follow exactly the idea behind Turing's proof of the Halting problem. Anyway, honestly, I would need to review his proof. It's been a long time since I read it. $\endgroup$
safety_programshould be able to return the correct answer (because this can easily be done for every finite program), but
hwill return the wrong answer, only because it inverts the output of the Oracle. So, you would say that safety is undecidable, while it actually is (if you use the definition above). $\endgroup$