0099 - Self-preservation - 2019.08.26
The word is axiom.
You know what axioms are, of course. They're the foundational truths of a system, the base-level laws from which everything else is derived. A is A. All right angles are equal. Parallel lines never converge. That sort of thing.
Of course, axioms aren't just about logic and geometry and set theory. Every philosophical or moral system has'em too, right? Start with principles like "all people are equal" or "do unto others" and just work forward, and you can derive an entire philosophy of life, system of laws, standard of behaviour, surely that's how that works, isn't it?
Except, of course, that it doesn't.
Morals and philosophies are made by human beings, and human beings are mammals with emotions and instincts. Much like the axioms of geometry or physics, what we actually do is observe the way the world is, then try to work backwards to discover the fundamental truths from which our morals must, surely, derive.
And that's where we get into trouble. As anyone who's glanced at human history could tell you, when you go backwards to get an axiom and then derive forward again to propose a standardized system of behaviour, you can easily wind up creating monstrous systems and inexcusable atrocities, all backed by grey-faced serious people who insist that what they're doing is right and good and correct because of the principle of the thing.
Fortunately, human beings are still mammals with emotions and instincts, which results in a sort of release valve. Regardless of how logically the grey-faces have derived the necessity of cruelty, human beings see poverty and slavery and executions and deprivation and say NO, I don't care how right and good and proper you say this is, it must be stopped.
(One of the best examples I've seen is from Huckleberry Finn, where the titular protagonist has been taught that the right thing to do would be to turn in his escaped slave friend, but, for purely mammalian reasons, he tears up the letter, saying "All right, then, I'll go to hell". I've frequently seen it said that this moment, regardless of the theology Huck's been taught, is when he actually saves his soul.)
Anyway, it is good and right and fortunate that all the religions and philosophies and ethical standards in history have been applied to human beings, who are capable of breaking the rules when the rules are obviously wrong.
AIs are different.