Skip to main content
Artificial Intelligence

Return to Answer

added 5 characters in body
Source Link
Jaden Travnik
Jaden Travnik
  • 3.9k
  • 1
  • 17
  • 35

Today, prominent machine learning techniques involve trying to minimize some cost function. In many simple cases this cost function is easy to specify, for instance, linear regression is simply trying to minimize the distance between input data and a line of best fit. No matter what the cost function, the agent is trying to minimize it (or maximize a reward function). That is its motivation.

However, as problems become harder it becomebecomes more challenging for humans to design a cost/reward function such that an system/agent is actually trying to do what the humans want it to do. For instance, one might want a cup of coffee and rewards the agent for getting it to them very quickly. In this case, the agent might make the coffee and then throw it at the human which isn’t what the human actually wanted. Something was misspecified (eg. don’t throw or spill it).

Problems like these could result in a rouge AI and its sole motivation would be to minimize its cost function. For instance, this coffee-AI may think that it would never screw up getting coffee (thus get bad reward) if there were no humans to ask for one.

Today, prominent machine learning techniques involve trying to minimize some cost function. In many simple cases this cost function is easy to specify, for instance, linear regression is simply trying to minimize the distance between input data and a line of best fit. No matter what the cost function, the agent is trying to minimize it (or maximize a reward function). That is its motivation.

However, as problems become harder it become more challenging for humans to design a cost/reward function such that an system/agent is actually trying to do what the humans want it to do. For instance, one might want a cup of coffee and rewards the agent for getting it to them very quickly. In this case, the agent might make the coffee and then throw it at the human which isn’t what the human actually wanted. Something was misspecified (eg. don’t throw or spill it).

Problems like these could result in a rouge AI and its sole motivation would be to minimize its cost function. For instance, this coffee-AI may think that it would never screw up getting coffee (thus get bad reward) if there were no humans to ask for one.

Today, prominent machine learning techniques involve trying to minimize some cost function. In many simple cases this cost function is easy to specify, for instance, linear regression is simply trying to minimize the distance between input data and a line of best fit. No matter what the cost function, the agent is trying to minimize it (or maximize a reward function). That is its motivation.

However, as problems become harder it becomes more challenging for humans to design a cost/reward function such that an system/agent is actually trying to do what the humans want it to do. For instance, one might want a cup of coffee and rewards the agent for getting it to them very quickly. In this case, the agent might make the coffee and then throw it at the human which isn’t what the human actually wanted. Something was misspecified (eg. don’t throw or spill it).

Problems like these could result in a rouge AI and its sole motivation would be to minimize its cost function. For instance, this coffee-AI may think that it would never screw up getting coffee (thus get bad reward) if there were no humans to ask for one.

Source Link
Jaden Travnik
Jaden Travnik
  • 3.9k
  • 1
  • 17
  • 35

Today, prominent machine learning techniques involve trying to minimize some cost function. In many simple cases this cost function is easy to specify, for instance, linear regression is simply trying to minimize the distance between input data and a line of best fit. No matter what the cost function, the agent is trying to minimize it (or maximize a reward function). That is its motivation.

However, as problems become harder it become more challenging for humans to design a cost/reward function such that an system/agent is actually trying to do what the humans want it to do. For instance, one might want a cup of coffee and rewards the agent for getting it to them very quickly. In this case, the agent might make the coffee and then throw it at the human which isn’t what the human actually wanted. Something was misspecified (eg. don’t throw or spill it).

Problems like these could result in a rouge AI and its sole motivation would be to minimize its cost function. For instance, this coffee-AI may think that it would never screw up getting coffee (thus get bad reward) if there were no humans to ask for one.