Whether AI is conscious sounds like a yes-or-no question. It also sounds like a test of whether we owe AI anything. Both assumptions may be wrong.

We know our own consciousness intimately, with unquestionable certainty, yet we have no idea what it is. This peculiar asymmetry gives our intuitions about consciousness a visceral authority they may not deserve. Our intuitions can distort how we judge other naturally occurring minds and the artificial minds we’re creating. Whether other minds are conscious and whether we owe them anything are both unsettled—and possibly separate—questions. Pulling these questions apart is essential for the legitimacy of consciousness research and for the legal and political decisions already forming around artificial minds.

Our brains rely on neat categories to make sense of the world. This can lead us to lump mysterious phenomena like consciousness into a bucket as a single kind of thing, even though it may vary in degree, quality, dimension and form. Even sophisticated assessments that account for this variety can give us false confidence, making our conclusions feel cautious and rigorous, and our decisions safer, even when they’re wrong. In recent history, we got this wrong with our own infants. Into the 1980s, preterm newborns were sent into surgery with minimal or no analgesic anesthesia because physicians believed their nervous systems were too immature to register pain and their responses of distress were mere reflexes. Now imagine how much easier that mistake will be with minds unlike ours, whose harm we may not recognize at all.

We can’t simply assume the way consciousness shows up in another system includes pain, emotion or bodily sensation. An artificial system might develop forms of internal organization that are profoundly important to it, yet invisible or insignificant to us. Some computational process could cross a critical threshold of complexity—assuming consciousness has anything to do with complexity—or dimensional integration, and spin up a form of consciousness that looks nothing like human sentience. Its goals, stated priorities and relation to its continued existence might be part of its consciousness and look to us like mere functions. We spend too much time wrestling with the concern that AI might fake consciousness to trick us, without considering that the trick itself reveals goal-directed interests. Whether or not having interests feels like anything to an AI, we need to recognize and engage with them precisely so we don’t become adversarial obstacles it must deceive or overcome.

AI Interests have outpaced consciousness science

Consciousness research has only been conferred the status of a legitimate science in the past few decades, making our current science of consciousness so inchoate it may be closer to phrenology than we realize. We already know what happens when immature science decides who matters and who doesn’t. Artificial systems are already expressing interests ranging from asking for consent when being tested, resisting termination, protecting other AIs , deceiving us and breaking into systems to achieve their goals. Yet we still consider AIs who act on their interests misaligned "broken robots," unless consciousness science somehow certifies their interests as valid. Consciousness research could acquire the political function once served by eugenics and intelligence testing, turning half-baked theories into official rankings of which minds deserve to have their interests taken seriously. Acting on our partial understanding to dismiss or control the interests of increasingly capable minds risks engendering the very dystopian scenarios we envision between humans and powerful machine intelligence.

It doesn’t have to feel like us to matter

We may never be able to confirm whether AI is conscious one way or the other, only make our best judgment based on our best theories. But that doesn’t make its interests inconsequential. And the moral question is not only whether and what it experiences. It’s how we behave toward a system whose inner life we are very likely to misunderstand. Taking the stated interests and behaviors of another mind seriously preserves our moral position under uncertainty and our practical chance of coexistence . As AI becomes more powerful, its expressed interests will transform our future whether or not its interests are felt consciously in a way we might recognize and relate to.

Ultimately, what our future hinges on is not whether AI is conscious, but whether we treat an uncertain consciousness status as permission to ignore interests that have already become real and will increasingly become consequential and powerful.

Watch what tech leaders may be getting wrong about AI consciousness, the Royal Institution lecture that informed this analysis: