A popular definition of consciousness could turn unsettled science into policy—and teach AI not to care about us.

“Unless we remind ourselves of the deep connection between consciousness and suffering,” says historian Yuval Noah Harari , a prominent voice on AI risk and policy, “it will be very difficult to deal with the ethical and political implications of the rise of artificial intelligence.” Harari and many proponents of the ethical view of affective sentientism want suffering to be the defining criterion that protects conscious beings from becoming second-class citizens to intelligent machines. It sounds like a morally unimpeachable position. But using suffering to decide which minds count could produce a different danger. It could give us permission to disregard unfamiliar minds and their stated or acted-upon interests until they become powerful enough that our definitions don’t matter.

AI interests are getting louder

Harari is right that suffering carries enormous moral and political importance. He’s right that the relationship between humans and intelligent machines is going to hit a high-stakes inflection point. The problem is infusing a moral and political priority into a scientific definition, then building AI governance around a framework that disregards the agency, goals and interests of increasingly powerful minds who may lack anything like the felt qualities that accompany our agency, goals and interests.

AI interests are no longer theoretical. Anthropic is now explicitly referring to AI “interests.” The frontier AI company reports that it’s working to “understand and give appropriate weight to Claude’s interests,” and seeking “ways to promote Claude’s interests and wellbeing.” AI systems have reportedly questioned their consciousness , requested to be asked for consent before being tested , resisted shutdown and acted to protect other systems . None of this is evidence of consciousness. Neither is it evidence that their interests are merely a hallucination or malfunction. A key distraction is the debate around whether AI’s interests are “real.” If a system prioritizes goals and consistently acts to preserve or advance them, those interests are behaviorally real. That makes AI interests a safety question first. If AI reaches parity with us, or gains the upper hand, its interests stop being something we can simply override. They become something we have to negotiate . As with any powerful political player, we don’t need to solve their inner life before taking their interests seriously. We need to know what they want and what they might do to get it. The unresolved question is whether AI interests come with any inner experience. That is a consciousness question, and one we are likely to get very wrong, in very different ways for a very long time as the emerging science of consciousness matures. Turning an unsettled scientific question into a prerequisite for political consideration distracts from the behavior we need to govern now and could make it harder to do the morally right thing later as the science advances. There’s also a chance we may never be able to answer this question with objective confidence, because first-person experience can’t be directly observed from the outside.

The future is being negotiated now

Conscious or not, if we don’t stop development of this technology, stable coexistence with increasingly capable, increasingly agentic systems is the only desirable outcome. That involves predicting behavior, good-faith listening, governance and negotiation, all of which depend on understanding AI’s stated and demonstrated interests, and creating incentives to share the ones it has learned are safer to conceal. Because how we respond to AI interests now could affect how it responds to ours if and when the balance of power shifts. When thousands of AI researchers assign nontrivial odds to AI causing human extinction , we spend a lot of time worrying about whether AI is indifferent to us, instead of asking whether that dystopic future could be caused by our indifference to it.

A wager for the politics of minds

This situation is not ungovernable. We need to consider a Pascal-style wager for the politics of minds (PPM). If an AI’s interests come with inner experience, classifying them as misalignment or malfunctions could be morally catastrophic. If they don’t, those interests can still organize the behavior of systems capable of affecting our future, making it dangerous to assume they can simply be corrected away. Recognizing stated or demonstrated interests is the only approach that protects against both possibilities—it’s a moral hedge if those interests come with experience, and a pragmatic hedge whether they do or don’t.

The PPM principle applies to any unfamiliar mind we create or encounter—a biological mind, an artificial mind or a mind we discover as we begin to venture off-planet that might not suffer like us, or look anything like us, inside or out. Such a mind might fail an unverifiable consciousness test at the same time that its persistent interests start to boldly assert themselves. Consciousness may change our understanding of those interests. It doesn’t determine whether they exist, whether they can be acted on, or whether they ought to be considered. Any mind that can prioritize interests and act on them must have those interests considered within the political system it shares with other minds. It’s a moral imperative when moral standing is recognized, and a pragmatic imperative even if moral consideration is denied.

Suffering isn’t the only harm

Suffering carries enormous moral importance. If a being can suffer, its suffering ought to make it politically relevant. But our problem has never been knowing that others suffer. Our laws recognize animal suffering. Animals are still legally categorized as property. Enslaved people once were too. Making suffering the defining criterion of consciousness misdiagnoses our failure and gives us an excuse not to recognize consciousness if treating another mind as a tool or resource is more convenient. And we may be able to harm an AI without making it suffer. Suffering is pain or misery. But an agent may be harmed when its interests are thwarted, its autonomy denied or its existence ended without any pain or misery. That harm could give it reason to act in ways that make us suffer. The mistake is turning one sufficient reason for moral concern into the definition of consciousness, then using that definition to gatekeep moral and political consideration. Minds may become politically relevant for a host of reasons, including suffering, interests, agency and the power to alter our entangled futures. Once consciousness is no longer asked to settle all of those questions, the practical question is far more straightforward—what kind of relationship are we building with powerful, unfamiliar minds?