There are times when a civilization hears the alarm and mistakes the sound for action.

We are living through one.

Jacob Coxon , after working at OpenAI and Anthropic, put the matter with brutal simplicity: “Do not underestimate the power of this technology.”

Power. That is the word we should refuse to pass over. Not performance. Not intelligence. Not productivity. Power.

Like Turing, Wiener, Good and Hawking before him, and Bostrom, Russell, Yudkowsky, Hinton, Bengio, Christiano, Carlsmith, Tegmark, Wozniak, Sutskever, Hassabis, Amodei, Leahy and, more recently, Gates, not without recalling the role popular culture has played in dramatizing such warnings through films such as Will Smith’s I, Robot , Coxon confronts that unsettling possibility: that intelligence may cease to be only a capability and become an autonomous means of acquiring “real power and resources,” ultimately posing an existential danger to humanity.

For three quarters of a century, 75 years from Wiener and Turing to the present day, prominent figures across science and technology have voiced different variations of the same underlying warning: intelligence, once made autonomous and superhuman, may cease to be simply a tool and become a form of power beyond human control.

Since then, almost everything about the artificial intelligence debate should have changed. There can be no excuse for not knowing, because our insatiable, almost Pavlovian appetite for warnings, which so often captivate us and now travel at nearly the speed of light, no longer affords us the luxury of ignorance.

But that is without reckoning with our disinterest in anything that might cause the annihilation of these warnings, by any solution capable of eliminating the conditions that give rise to them and preventing the cascade of this stimulus, never innovative because always the same, which makes it possible to buy attention cheaply, at scale, recyclable ad infinitum, while such solutions are carefully kept in a perfectly opposed and equidistant position from our constant taste for warning.

So we can keep waiting, and expect yet another avalanche of headlines to carry the same warning soon enough, reopening the same questions, provoking the same astonishment, the same hesitation, the same grave expressions of concern, while we remain the mouse trapped inside the glass jar, condemned to repeat the same gestures, retrace the same path, and run in circles as if repetition itself were a form of progress.

Or we can escape the cycle and get this.

A technology that calculates faster is an engineering problem. A technology capable of accumulating and exercising power is a political problem. And a political problem of sufficient magnitude eventually becomes a constitutional one.

Humanity has encountered this problem before. Kings assured us that power was safe in wise hands. Empires assured us that power was safe in civilized hands. Institutions assured us that power was safe in expert hands. History’s answer has been remarkably consistent. The character of the ruler is not a substitute for the architecture of power. We did not invent constitutional government because every ruler was wicked. We invented it because even good rulers are human, successors change, interests collide, and concentrated power eventually learns to justify itself.

Why should algorithmic power be granted an exemption from one of civilization’s hardest-won lessons?

Whether artificial intelligence is, or will be, safe narrows our field of vision to questions that are predominantly utilitarian in focus. That power should be safe is useful, even necessary. But our need for safety and certainty must not impair our ability to see that utility itself confers power. And when that power becomes great, a question more important still than its usefulness arises. That of the legitimacy of its exercise.

What makes algorithmic power legitimate?

To what ends should algorithmic power be created, and who has the authority to define those ends? This is the legislative-like question we do not ask.

Who should be permitted to exercise algorithmic power, under what mandate, and within what limits? This is the executive-like question we do not ask.

Who should possess the independent authority to evaluate whether its exercise is fair, just, lawful and acceptable, and by what standards should it be judged? This is the judicial-like question we do not ask.

And finally, how should the authority of algorithmic counterpower be constituted: the power to contest, restrain, suspend, reverse or revoke the exercise of algorithmic power itself? This is the constitutional question we are not asking.

This is the premise of Trias Algorithmica 1 .

Where consequential algorithmic power exists, the authority to define its rules, execute them and judge their legitimacy must not collapse into the same hands.

Not because every AI company is malicious. Because no institution should be asked to be legislator, executive and judge of its own power.

The people within those institutions may understand the civilizational danger of great unchecked power. Because they may be intelligent. Because they may be responsible. Because they may be sincere. Because they may even desperately want a safe outcome. But still, they will drive this quest for the advance of such a consequential power forward.

This destroys one of the great comforting illusions of our age. That responsible people will solve the problem by behaving responsibly.

If Company A slows its AI race down. It fears Company B will not. Company B fears Company C. Company C fears Company A. And loop is closed. Each can defend acceleration as an act of responsibility. The absurdity is perfect.

Everyone races towards a danger precisely because everyone fears the danger of the race.

If that logic sounds irrational, it is not. It is rational behavior inside a badly designed system.

You cannot base civilization’s safety upon competitors voluntarily refusing advantages they believe others will seize.

When responsible actors are systematically pushed toward outcomes they themselves fear, the failure is no longer primarily moral. It is institutional.

Many of those issuing these warnings have recently contemplated something as drastic as a temporary ban on improving model capabilities. Whether one supports such a ban is almost secondary.

Its very appearance tells us how impoverished the debate has become and why it should not remain so.

We have reduced our imagination to two buttons: Accelerate. Stop. Build everything. Or build nothing.

But civilization has encountered powerful technologies and institutions before without choosing permanently between surrender and prohibition.

We invented a third possibility: govern power by separating power.

A pause might buy time. But time for what?

If tomorrow’s moratorium ends with precisely the same companies retaining the same concentrated authority to self-build, self-deploy, self-evaluate and self-authorize increasingly consequential algorithmic systems to enter society, then we have delayed the question, not answered it.

Trias Algorithmica 1 therefore asks what the pause debate does not.

What determines the permissible objectives of algorithmic power? What authority controls the infrastructure through which that power is exercised? What mechanism authorizes the deployment of consequential algorithmic systems? What independent structure evaluates whether those systems are fair, lawful, acceptable and safe? What power can suspend or stop their operation? What mechanism allows their decisions to be challenged or appealed? What counterpower can say no to those who possess the machine?

These are the anatomy of freedom in an algorithmically mediated civilization.

Not another ethics charter written by those it is supposed to constrain. Not another safety committee dependent upon the organization whose systems it evaluates. Not another promise that unprecedented concentration of power will remain benevolent because its current custodians understand the risks.

We need something older, harder: counterpower.

And never construct an extraordinary system in which civilization depends upon extraordinary trust in the first place.

That the AI race warning is once again becoming visible is not a good sign; we have been seeing these warnings repeatedly for at least seventy-five years.

We are forced to acknowledge that an alarm tells us the house may burn. But once its shrill sound falls silent, everyone behaves as though silencing the alarm had somehow extinguished the possibility of fire.

The race that matters now is therefore larger than the race toward artificial intelligence and wiser than the race to stop it.

It is the race to establish the conditions under which current algorithmic power can exist without becoming absolute power, before greater capability hardens greater algorithmic power into ever more absolute power over people’s lives.

The warnings go mainstream.

Now the question of what makes algorithmic power legitimate must go mainstream too.

1 Learn more in, Trias Algorithmica: What Code Rules.