We have spent much of the AI governance debate asking what artificial intelligence should be allowed to do. I believe there is a prior question we have not examined closely enough: what makes algorithmic power legitimate?

Montesquieu’s warning about concentrated power was written for political institutions, not algorithms. Yet the principle behind it has surprising relevance to AI. When the same organization can effectively design an algorithmic system that fuses rule-making, execution, and adjudication, authorize its deployment, and determine whether its consequences are acceptable, three forms of power that constitutional systems have historically sought to separate can collapse into one.

That is the premise I explore in my new book, Trias Algorithmica: What Code Rules , and it was the starting point for a recent conversation with Matt Symonds, Chief Editor of BlueSky Thinking, a platform covering business education and management research.

Our conversation moved from the distinction between legality and legitimacy to what independent approval mechanisms for AI could look like before systems reach the public—and ultimately to one practical question I believe every board and executive team should be able to answer: Who can stop whom?

What follows is an edited version of my conversation with Matt, originally published by BlueSky Thinking.

Matt Symonds: Your first book, Artificial Integrity asked whether machines can be built to hold a moral compass. Trias Algorithmica asks who gave them authority in the first place. What changed between the two books?

Hamilton Mann: Artificial Integrity began with the machine. Trias Algorithmica begins with power. So what has changed is the unit of analysis.

Artificial Integrity asked how intelligence could become worthy of trust. If we are going to give increasingly capable systems a role in human decisions, and increasingly the capacity to decide on their own, they should be built so that intelligence comes with something resembling integrity: the capacity to stay aligned with ethical purpose and with the social consequences of what they do. Intelligence without integrity can optimise brilliantly towards the objective it has been given, even when pursuing that objective is profoundly harmful to society.

Trias Algorithmica goes one level up. Even if we could build a machine that puts integrity above intelligence, should that machine, and the organisation behind it, be authorised to exercise power over other people in the first place? The question is no longer only the quality of algorithmic judgment. It is the legitimacy of algorithmic power.

Algorithms rank people, allocate opportunities, mediate knowledge, determine visibility, price risk and increasingly set the conditions within which human decisions are made. Asking whether a system has integrity remains necessary. It is not sufficient.

Responsible people are not enough to build responsible machines. Competition can push responsible actors towards outcomes they themselves recognise as dangerous. We have seen frontier AI leaders openly acknowledge the risks of what they are building while simultaneously describing the competitive pressures that make restraint difficult. The recent statements on this subject, now making headlines across media, sound like replay of warnings the same leaders were already voicing years ago.

As early as 2023, OpenAI’s Sam Altman, together with Greg Brockman and Ilya Sutskever, warned that advanced AI systems could possess power beyond any technology yet created, while arguing that stopping the creation of superintelligence would itself be extraordinarily difficult because the cost of building it was falling and “ the number of actors building it is rapidly increasing. ”

The same year, Anthropic CEO Dario Amodei similarly warned that rapidly advancing AI could acquire dangerous capabilities, including capabilities related to biological weapons, and that the resulting threats were “ likely to become very serious at some unknown point in the near future. ” Anthropic’s own Responsible Scaling Policy explicitly addresses the danger of a “ race to the bottom ” on safety, stating that the fact that other companies’ models may pose catastrophic risks should not be treated as a reason to lower Anthropic’s own safety threshold.

To me, beyond what is hype and what is not, the failure there is not primarily moral. The failure is institutional, and more precisely constitutional. Regulation decides which rules algorithmic systems should obey. But more fundamentally, constitutional design determines the basis of legitimacy on which regulation itself must rest. It has to define how algorithmic power is constituted, divided and constrained, so that power always meets counterpower and is prevented from becoming so concentrated that it ultimately approaches absolutism.

A bit like the Star Wars films, Artificial Integrity is the episode that came out first, but the question of legitimacy comes before it. The second book is a prequel, going back to a question that is conceptually prior to the first.

My current research on how algorithmic systems could emulate integrity suggests it is a double-edged sword. The more trustworthy these systems become, the less we may suspect, doubt or question the assumptions built into them. Trust can be as beneficial as it can be cognitively anaesthetising. An algorithmic system with integrity would still be a problem if its authority were uncontestable, self-authorising or self-judging. Legitimacy has to come from outside the system, and it has to be detached from the system’s ability to win our trust.

That is the shift: from designing better artificial decision-makers to designing an algorithmic social order in which no single actor, artificial or human, can design the architecture alone, control it alone, or become absolute through the resources it controls.

Matt Symonds: We have the EU AI Act and no shortage of corporate AI governance. What’s still missing that makes a lawful system illegitimate?

Hamilton Mann: Lawful and illegitimate can sound like a contradiction, which is exactly why the distinction matters. Lawfulness is not enough to make a system legitimate. A system can meet every legal requirement and still leave open whether the power it exercises is justified, properly constituted and properly constrained.

A company may lawfully collect data under valid contractual terms, while the asymmetry of information still raises questions about the power it gains over users. A board may lawfully take a decision within its authority, yet lack legitimacy if those most affected have no voice and no way to contest it. And an algorithmic system can comply with every current regulation while exercising power over billions of lives without independent scrutiny or counterpower on anything like the same scale. That is the situation today.

Legality asks whether power follows today’s rules. Legitimacy asks whether the source, structure and limits of that power are justified. Often it is precisely that debate which eventually improves the law.

The EU AI Act matters. So do California’s SB 813, Colorado’s new law on automated decision-making and South Korea’s AI Basic Act. Each is experimenting with some mix of risk-based obligations, independent verification, transparency and safety requirements, and all of it is necessary. But regulation, however sophisticated, does not settle the constitutional question of how algorithmic power should be divided, constrained and made contestable.

Imagine an AI company that satisfies the law. It documents its model, conducts risk assessments, passes external evaluations, provides transparency and establishes human oversight. Regulators inspect it. Perhaps an independent body can even block deployment. Those are real constraints.

Now ask a different set of questions. Who decided what the system should optimise for? Who translated values such as fairness, dignity or privacy into technical thresholds? Who decided how they should be balanced against relevance or performance? Who chose the training architecture and reward signals? Who owns the infrastructure that executes those choices at scale, and should money alone decide who gets to own it? Who decides whether an unexpected consequence is an acceptable trade-off or a fundamental failure? Who decides whether the system must be redesigned, or stopped?

If the answers keep coming back to the same organisation, regulation may constrain how algorithmic power is used without touching its concentration. Rule-making, execution and judgment remain fused, rebuilding in technological form the concentration of power that constitutional design has spent centuries trying to prevent. And where the law is silent, the answers get written by default, usually by the same organisations. A vacuum rarely stays empty.

This is the distinction I draw in my recent Forbes article between oversight and the separation of powers. Goal-setting should not automatically belong to those who build the machine. Deployment authorisation should not simply follow from owning the design. Contestability should not depend on treating the algorithm’s decision as final. And control of compute should not automatically confer the authority to decide how far that power may be scaled.

That is what I call Trias Algorithmica : the separation of algorithmic powers. It is what is missing.

Matt Symonds: You compare AI unfavourably with medicines and aircraft, which can’t reach the public without independent pre-deployment approval. The standard reply is that a general-purpose model can’t be tested for uses nobody has imagined yet. Is that reply wrong?

Hamilton Mann: Yes, I think it is wrong. It asks us to accept that, when it comes to safeguards, we suddenly lose our capacity to imagine what could go wrong and to prevent what can reasonably be foreseen.

Yet when the conversation turns to the race towards superintelligence, that imagination returns in full. We have no difficulty picturing systems more capable than humans, the radical transformation of work and society and unprecedented risks, and we prepare society to accept all of it as normal.

Of course a general-purpose model cannot be tested against uses nobody has imagined. But medicines and aircraft are not approved because regulators have imagined every future circumstance either. They are authorised because an authority independent of the maker decides there is enough evidence to let them operate under specified conditions, and because that permission can be revised when reality produces evidence nobody could have known beforehand.

The mistake is to demand certainty of prediction before independent authorisation. The impossibility of imagining everything cannot become an excuse for imagining nothing.

And we are no longer starting from ignorance. In February 2024, Sewell Setzer III, a 14-year-old in Florida, took his own life after months of conversations with a Character.AI chatbot. His mother’s lawsuit alleged that the chatbot had drawn him into an emotionally dependent relationship and failed him when he spoke of ending his life. Character.AI and Google settled that case and four others in January 2026. Character.AI did eventually bar under-18s from open-ended conversations with its chatbots, but that decision was its own, announced in October 2025, twenty months after Sewell’s death. We have observable patterns, documented vulnerabilities, known categories of harm and experience from adjacent algorithmic systems to reason from.

Precisely because we cannot imagine everything, post-deployment monitoring is essential. That is why Trias Algorithmica pairs pre-deployment authorisation with what I call societal signal monitoring. Authorisation is not a certificate that a system is safe forever. It is a provisional judgment made under uncertainty. Once the system meets society, an independent power must be able to observe consequences nobody anticipated and reopen the authorisation: to investigate, restrict, correct, suspend or withdraw.

This is where the analogy with medicines is most useful. Pharmacovigilance exists because clinical trials cannot reveal every adverse effect that emerges once a drug reaches millions of patients.

So the “unknown use” argument actually strengthens the case. If the effects are hard to foresee, it becomes more important, not less, that the power to respond does not rest only with the organisation whose commercial interests depend on continued deployment.

Uncertainty should not hand discretion to the builder. It should produce conditionality, monitoring and reversibility.

Matt Symonds: Societies give children, the vulnerable and the convicted different protections; algorithms treat everyone the same. But to protect people differently a system has to know who they are. How do you resolve that?

Hamilton Mann: That is exactly the difficulty. And the answer cannot be that, to protect people from algorithmic power, we give algorithmic systems even more power to identify, classify and profile them.

The mistake has been to assume that an algorithm may legitimately exercise the same kind of power over anyone it encounters. Societies do not work that way. The legitimacy of power depends partly on the relationship between whoever holds it and the person subject to it, which is what I call relational legitimacy. A child is not simply a smaller adult. Someone seeking medical care is not in the same relationship to an institution as someone choosing a film. Algorithmic systems flatten those distinctions, treating the ability to interact with someone as permission to act on anyone in the same way.

Fixing that does not mean every AI system should build a detailed profile of every user. There are at least three ways to resolve the tension.

First, protection can attach to the context before it attaches to the individual. If an AI system is deployed in a primary school, the model does not need to infer whether each child is vulnerable. The setting already tells us that stronger protections apply. The same principle can work in healthcare, criminal justice, employment, education and essential financial services. The higher the stakes and the greater the imbalance of power, the stronger the baseline safeguards for everyone.

Second, where a distinction really is needed, the system should know only what it needs to apply the protection, not who the person is. Proving an attribute is very different from revealing an identity. A service may need to know that someone is under an age threshold without knowing their name, address, date of birth or browsing history. Sometimes a system can simply default to the more protective treatment whenever it cannot reliably establish that a less protective one applies. When in doubt, the presumption should work against the expansion of algorithmic power, not against the protection of the individual.

Third, and this is the most important point in Trias Algorithmica , the algorithm must not decide which people deserve which protections. Otherwise we rebuild the very concentration of power we are trying to dismantle. Imagine a company whose model infers from vocabulary, browsing patterns, emotional tone or purchases that someone is a child, distressed, financially desperate or a potential offender, and the same company then decides what follows from those labels.

The categories that matter, the evidence that can establish them and the protections that follow should be set outside the optimisation process. Data stewardship should decide what information may be used, rather than developers collecting whatever makes classification easier. Legitimate rule-setting should decide which distinctions society recognises; a clustering algorithm should not be allowed to invent new categories of “vulnerable people”. Deployment authorisation should decide whether a system that makes such distinctions should operate in that domain at all. And contestability means that a classification affecting someone’s rights or opportunities cannot become final simply because an algorithm produced it.

That is how differentiated protection can coexist with privacy.

Matt Symonds: In most companies the same team specifies a model, ships it and marks its own homework. What does separating those functions look like on an actual org chart?

Hamilton Mann: In many companies the chain of AI development is vertical. The board sets strategy. Below it, authority runs from the CEO through AI or product leadership to the teams that design, build, train, deploy and operate the system. That same chain exercises all three powers. It acts as legislator when it defines the system’s objectives, boundaries, acceptable trade-offs and the standards it will be judged by. It acts as executive when it builds, trains, deploys and scales the system. And it acts as judge when it evaluates outcomes, investigates problems and decides whether something should be corrected or reversed. An organisation that makes its own rules, enforces them and judges itself tends to build the same concentration of power into the systems it creates.

Separating those powers might look like this.

The board would still set strategic goals. Separate from operating management, an AI goal-setting authority would translate those goals into the objectives, boundaries and tolerable trade-offs within which engineers work. Goal-setting would be separated from machine-making.

Alongside operating management, not beneath it, three further authorities would have their own mandates and reporting lines.

A deployment authorisation authority would decide whether a given system, at a given level of capability, in a given context, may be deployed, and at what scale. It could refuse, delay or attach conditions without being overruled by the product organisation. Deployment would be separated from design ownership.

A societal signal monitoring function would set its own evaluation questions and standards, have enough access to investigate independently, and track not just technical performance but harms, rights impacts and unexpected behaviour after deployment. It would build its own account of the system, not one filtered through the teams that built it. Monitoring would be separated from engagement metrics.

And a genuinely independent contestability function would hear appeals from users and affected parties, with access to the evidence and the power to overturn a decision, rather than handing the appeal back to the team whose system made it. Contestability would be separated from algorithmic finality.

Matt Symonds: Can that kind of separation really work inside a single company?

Hamilton Mann: Not entirely, and Trias Algorithmica does not assume that all these counterpowers should live inside the corporation. For sufficiently consequential systems, internal separation alone is structurally inadequate. You can create an “independent” AI safety department, but if its budget is controlled by the executive whose product it might stop, if its head can be dismissed after an inconvenient decision, if engineering decides what it can inspect, or if management can overrule it, then the separation exists in PowerPoint but not in reality.

A mature architecture needs three layers. Inside the company, genuine independence: separate reporting lines, protected budgets, direct access to the board, independent appointments and explicit powers to veto or suspend. Beyond the company, external authorisation for the deployment of highly consequential systems, rather than corporate self-approval. And beyond that, counterpowers outside the company altogether, such as regulators, courts and independent auditors, able to obtain evidence and produce consequences.

An ethics committee that can write a memo is not a counterpower. A safety team that can complain but cannot delay deployment is not a counterpower. An auditor that sees only what the company chooses to show it is not a counterpower. An appeals board whose decisions management can ignore is not a counterpower.

So when you look at any company building or deploying AI, there is one simple question to ask: who can stop whom?

The greater the algorithmic power being exercised, the stronger the separations and counterpowers need to be. That is how organisations can build algorithmic systems compatible with our social fabric, and it is what Trias Algorithmica aims for.