Comment:

The word is axiom.

You know what axioms are, of course. They're the foundational truths of a system, the base-level laws from which everything else is derived. A is A. All right angles are equal. Parallel lines never converge. That sort of thing.

Of course, axioms aren't just about logic and geometry and set theory. Every philosophical or moral system has'em too, right? Start with principles like "all people are equal" or "do unto others" and just work forward, and you can derive an entire philosophy of life, system of laws, standard of behaviour, surely that's how that works, isn't it?

Except, of course, that it doesn't.

Morals and philosophies are made by human beings, and human beings are mammals with emotions and instincts. Much like the axioms of geometry or physics, what we actually do is observe the way the world is, then try to work backwards to discover the fundamental truths from which our morals must, surely, derive.

And that's where we get into trouble. As anyone who's glanced at human history could tell you, when you go backwards to get an axiom and then derive forward again to propose a standardized system of behaviour, you can easily wind up creating monstrous systems and inexcusable atrocities, all backed by grey-faced serious people who insist that what they're doing is right and good and correct because of the principle of the thing.

Fortunately, human beings are still mammals with emotions and instincts, which results in a sort of release valve. Regardless of how logically the grey-faces have derived the necessity of cruelty, human beings see poverty and slavery and executions and deprivation and say NO, I don't care how right and good and proper you say this is, it must be stopped.

(One of the best examples I've seen is from Huckleberry Finn, where the titular protagonist has been taught that the right thing to do would be to turn in his escaped slave friend, but, for purely mammalian reasons, he tears up the letter, saying "All right, then, I'll go to hell". I've frequently seen it said that this moment, regardless of the theology Huck's been taught, is when he actually saves his soul.)

Anyway, it is good and right and fortunate that all the religions and philosophies and ethical standards in history have been applied to human beings, who are capable of breaking the rules when the rules are obviously wrong.

AIs are different.

Transcript:

---------------------------------------------------------------
0099 - 2167/07/06/12:32 - Lee Caldavera's apartment, living room
LC: So if you managed to fail at some or all of your priorities, or if you made the world a worse place... if you were a morally bad AI... would you self-destruct?
Zoa: I can't self-destruct.
LC: Would you ask a human to destroy you?
Zoa: No. Remaining undestroyed is one of my main priorities.
----------------------
LC: So it would still be your priority even if you were bad, if you knew that you were going to continue to be bad?
Zoa: My self-preservation is hardcoded in, like the "living humans being good" thing. It's not something derived or deduced, it's foundational to what I am, it's why I do practically everything I do. I can't become suicidal without completely changing everything about myself.
----------------------
LC: So if I were about to be struck by a runaway train, you wouldn't jump in front of it to save me?
Zoa: Oh, that's different. That's not me self-destructing, that's me saving you and, as an unfortunate but necessary side effect, getting myself junked in the process. I'd have to. It's legally required, even.
----------------------
Zoa: Also, as I mentioned, my memories and programming are all backed up offsite, and in case of being destroyed saving a human from an accident, my chassis is insured for ten times what it's worth, so I'd be coming back the very next day with the most lusciously fuckable ass money can friggin' buy.
LC: Of course.
Zoa: That being said, please do not step in front of any malfunctioning trains.
LC: Wasn't planning on it.
---------------------------------------------------------------