Does this prove AI Safety is undecidable?
Output meaning output to computer program.
[A1] Assume we have a program that decides which outputs are “safe”.
[A2] Assume we have an example of an unsafe output: “unsafe_output”
[A3] Assume we have an example of safe output: “safe_output”.
[A4] Define a program to be safe if it always produces safe output.
[A5] Assume we have a second program (
safety_program) that decides which programs are safe.
[A6] Write the following program:
def h() h_is_safe := safety_program(h) if (h_is_safe): print unsafe_output else: print safe_output
Clearly h halts.
If the safety_program said h was safe, then h prints out unsafe_output.
If the safety_program said h was not safe, then h prints out safe_output.
Therefore safety_program doesn’t decide h correctly.
This is a contradiction. Therefore we made a wrong assumption: Either safe output cannot be decided, or safe programs cannot be decided.
Therefore, in general, the safety of computer programs, including Artificial Intelligence, is undecidable.
Therefore AI Safety is undecidable.