Does this prove AI Safety is undecidable?

Question

Does this prove AI Safety is undecidable?

Proof:

Output meaning output to computer program.

[A1] Assume we have a program that decides which outputs are “safe”.

[A2] Assume we have an example of an unsafe output: “unsafe_output”

[A3] Assume we have an example of safe output: “safe_output”.

[A4] Define a program to be safe if it always produces safe output.

[A5] Assume we have a second program (safety_program) that decides which programs are safe.

[A6] Write the following program:

def h()
   h_is_safe := safety_program(h)
   if (h_is_safe):
      print unsafe_output
   else:
      print safe_output

Clearly h halts.

If the safety_program said h was safe, then h prints out unsafe_output.

If the safety_program said h was not safe, then h prints out safe_output.

Therefore safety_program doesn’t decide h correctly.

This is a contradiction. Therefore we made a wrong assumption: Either safe output cannot be decided, or safe programs cannot be decided.

Therefore, in general, the safety of computer programs, including Artificial Intelligence, is undecidable.

Therefore AI Safety is undecidable.

Answer 1

You are essentially correct, although there may be some minor holes in the structure of your argument specifically. But if we're speaking informally, yes, you are correct. This is a consequence of Rice's Theorem. And it isn't just true for AI safety. All non-trivial semantic properties of algorithms (or functions) are undecidable.

"Semantic" just means it is about what the program outputs, or its behavior, both used synonymously to mean the actual result of the function, not syntax, aka the source code: the function specification itself. "Nontrivial" means that it isn't true for all programs or false for all programs.

Answer 2

In my opinion, there are several flaws in your proof and reasonings.

First, note that, in the case of Turing's proof, h will actually loop forever (i.e. not halt) when the oracle says that h halts. In this case, there's an actual contradiction, because h will do the opposite of what the oracle says.

So, to follow Turing's proof, you would need to make h behave unsafely if the oracle says h is safe. But how should we define a safe or unsafe program? There are many unsafe behaviors. For example, in a certain context, an insult could be unsafe, in other contexts, a certain limb movement could be unsafe, and so on. So, an agent is unsafe or behaves unsafely usually with respect to another agent (or itself) or environment. You probably need to keep this in mind if you want to prove anything about the safety of AI agents.

In your second assumption, you are implicitly saying that any machine that produces the output unsafe_output is unsafe, but, of course, this definition is not a realistic definition of an unsafe program.

To help you define safety in a more reasonable and natural way, I think it may be useful to reason first in terms of artificial agents, which are higher-level concepts than Turing machines. Then you could find a way of mapping agents to TMs and attempt to prove your conjectures by using the tools of the theory of computation.