Are we supposed to "decide" ethics for intelligent systems?

Question

Ethics is defined to be the set of moral principles that governs a person's or group's behavior.

Innately, shouldn't any system devise ethics for itself? Most of our articulations, in either direct or indirect manner, talk about intelligent systems in context of human presence. So, why, in first place, are we studying AGI ethics?

Given Roseau's reasoning for a society

Human beings can improve only when they leave the state of nature and enter a civil society.

Doesn't that also automatically apply to any intelligent system? If it does, then can't we imply that any inorganic intelligent system should, perhaps, come as an offset of a network of intelligent computers (a society per se)?

Who are we to constitute ethics and rules, morale for another society which, perhaps, could be much more intelligent than us?

Answer 1

Nonhuman ethics can be totally arbitrary

Regarding your citation, perhaps it's useful to consider why should anything that philosophers have said about human ethics apply for arbitrary nonhuman systems? We all share a common biological background, our behavior, emotions and especially positive/negative feelings arise from the particular way how our brains are built. Man is by nature a social animal, and much of our behavior and feelings of empathy, compassion, fairness, greed, social status, etc are formed and expressed in society. Our minds are adjusted for this, as are the minds of other mammals. A human that forms an ethical system when joining a civil society does so in the basis of all this shared context, and within the limitations of it.

However, all this is orthogonal to intelligence. All the variety of mankind encompasses a tiny island in the whole space of possible intelligences and their goals. Any system of ethics is feasible, a vast majority of them having no relationship whatsoever to what we could consider ethical. A human can be expected to join a society and learn ethics because our brain is hardwired to respond to social stimuli - an AI would learn ethics from a society only if its previous ethics already happen to value that highly, and that isn't likely to happen without careful design.

There is a thought experiment that is commonly used as an example - the "paperclip maximizer", an artificial agent with an ethical system that defines 'good' proportionally to the number of paperclips that exist in the universe and only that - and that is a perfectly reasonable example, because a completely random ethical system is overwhelmingly likely to reduce to something stupid like that. The space of "reasonable" (to us) ethical systems is very tiny compared to all ethical systems possible, and expecting that a randomly selected ethical system will just happen to land there is like relying on winning a high-stakes lottery.

We want their ethics to contain certain things

So the main reason why we are studying AGI ethics is because we (or some of us) really, really, want the AGI ethical system to include certain things. For example, I'd like that ethical system to include my survival. I'd prefer that ethical system to be consistent with my happiness and wellbeing - and the very existence of a very powerful entity that simply happens to not care about me at all is not compatible with my survival and wellbeing, so I'd really prefer for it to care about me, or alternatively never become very powerful.

One could certainly argue that we have no right to constitute ethics, rules and morale for another society which could be much more intelligent and better than us. That another society might be "better" in some manner - but better for whom ? If it's better for "them" but worse for me (and humanity), then I have strong motivation to simply claim that right and try my hardest to ensure that this hypothetical future society includes me, my desires and my ethics.