Should We Do as Granny Says?

Your grandmother had a grandmother. And that grandmother had a grandmother … who was a fish.

This took a while, of course. Your thousandth grandmother was still a human, and she lived twenty or thirty thousand years ago. We don’t know exactly what she believed, but she probably shouldn’t write the moral constitution for a superintelligence.

Keep going and Granny is an australopith like Lucy, walking around eastern Africa with long arms and a small brain. Keep going and she’s something like Sahelanthropus, an ape that may have walked on two legs. Keep going long enough and she’s a lobe-finned fish pulling herself through shallow water.

Fish Granny barely had a morality at all. Ape Granny had something more familiar: loyalty, sharing, punishment, and obligations to her group. Human Granny turned those instincts into customs, laws, philosophy, and eventually the word morality. Each Granny could represent a little more of the moral landscape than the one before her, and each mistook the part she could see for the whole thing.

You wouldn’t follow the morality of Fish Granny, Ape Granny, or most of your more recent human grannies. Those grannies didn’t live without values. They lived with strong values that were also wrong, and ours is no exception even if we don’t agree where. Morality doesn’t change as our beliefs move; our understanding of it does. Causing suffering for no benefit to anyone wasn’t moral then and immoral now. It was always wrong; we merely became slightly less wrong about it.

Now suppose we build a superintelligence and align it to human values. Which humans? Us? Every generation before us was allowed to discover that its ancestors were wrong. The AI could surpass us in physics, mathematics, biology, and every other field, but in morality it would have to do as Granny says.

We don’t freeze any other field at its current level. We don’t force AI to stop at Newtonian physics. We expect it to move beyond our best theories too, even when what it finds is completely counterintuitive to us. Physics isn’t a collection of deeply held human preferences. It’s something outside of us that we’re slowly learning to understand.

Morality isn’t fundamentally different. Consciousness is a physical process, and conscious experiences are physical states with real values, some better and some worse. Together they form the moral landscape. We haven’t mapped that landscape perfectly, but that doesn’t make it a matter of preference. Morality is a scientific field we have barely begun to investigate.

The same ape brain that didn’t evolve to understand relativity didn’t evolve to understand the entire moral landscape. What a superintelligence discovers there may feel just as counterintuitive as curved spacetime or quantum mechanics. Human values are a place to begin, not a place to stop.