For a brief moment, Facebook seemed to have discovered how to separate its own power. But it had not. Chances are, no other will do so by itself.

In 2020, after years of criticism over its power to decide what billions of people could say and see, the company launched an independent Oversight Board . Mark Zuckerberg had earlier imagined such a body as “almost like a Supreme Court,” composed of people outside Facebook who could independently review its decisions. Facebook itself explained the rationale more plainly: “Facebook should not make so many important decisions about free expression and safety on our own.”

The Board could hear selected appeals after users exhausted Meta’s internal process, and its decisions in those cases could bind the company. It reversed some of Facebook’s content decisions. It forced explanations. Its recommendations helped change policies and enforcement practices. For a technology company accustomed to writing its own rules and policing them itself, this looked like an unusual surrender of authority.

And yet, in August 2026, Meta was facing one of the most sweeping legal challenges ever mounted against a technology company over allegations that Facebook and Instagram had been designed to addict children and that the company had misled the public about their safety. The resulting multistate agreement was described by state attorneys general as the largest state consumer-protection settlement in history outside the Big Tobacco settlements. Meta denied wrongdoing but agreed to a proposed settlement worth up to $18 billion , along with sweeping changes to how its platforms serve young users.

That raises an awkward question: Facebook had operated its “Supreme Court” for six years. Where was it? And why was an institution created to check Facebook’s power unable to prevent or meaningfully constrain the practices at the heart of a controversy of such scale?

The answer is banal but revealing: the Board was exactly where Meta had put it. It could judge content decisions, while the architecture generating those decisions remained outside its jurisdiction. Meta had separated judgment without surrendering the power to decide what could be judged.

Interestingly, the Board was sounding alarms from inside a constitutional structure whose limits Meta itself had drawn.

In 2023, three years before the settlement, the Board considered two Facebook videos promoting an extreme fruit-juice diet . The videos had attracted more than two million views and were monetized. In examining them, the Board explicitly pointed to research showing that social-media recommender algorithms can lead adolescents toward increasingly extreme diet-related content and warned that adolescent girls are particularly vulnerable to eating disorders and related harms. It recommended that Meta stop providing financial incentives for extreme and harmful diet content. Some Board members wanted Meta to go further: restrict such content to adults and attach health warnings.

The Board continued to confront children’s rights. In 2024, it pressed Meta to strengthen its rules on content facilitating child marriage . In 2025, it examined how Facebook should handle videos depicting child abuse , balancing the protection of children against the need to expose abuse. That same year, when assessing Meta’s changes to its Hateful Conduct rules, it specifically asked the company to investigate how those changes might harm LGBTQIA+ people, including minors , and to report publicly on mitigation efforts every six months. By February 2026, the Board had made young people’s rights and wellbeing an explicit strategic priority . In May, months before the settlement, it began a separate project examining how AI chatbots should be governed for users aged 13 to 17 .

So the problem was not simply that nobody saw the warning signs. The deeper problem was that the institution capable of identifying them did not possess the power to compel changes to the architecture producing them.

It could tell Meta that monetization might encourage harmful material. It could warn about recommender algorithms. It could ask for stronger safeguards for minors. It could demand explanations and recommend policy changes. But it could not compel Meta to redesign the recommendation engine. It could not impose a two-hour limit on teenagers. It could not switch off notifications during school hours. It could not require a non-personalized feed. It could not independently determine the conditions under which Meta’s products were allowed to engage children.

The Board’s recommendations remained just that: recommendations. Under its governing rules , decisions on individual pieces of content could bind Meta; broader policy recommendations could not.

It could overturn a decision. It could not overturn the system that kept producing the decisions.

Two days after the settlement, the Oversight Board itself appeared to acknowledge the structural problem. Writing about teen safety , it argued that “real leadership here means inviting scrutiny that companies don’t control. That means independent and external oversight capable of making decisions based on the rights of people, rather than the commercial interest of the business.” The irony is difficult to miss. Meta had already created an independent oversight institution. The settlement exposed the deeper problem: independence without sufficient authority is not enough.

That is what makes the eventual settlement so revealing. And this exposes a misunderstanding that extends far beyond Facebook.

We have confused oversight with separation of powers.

A court, however independent, does not create a separation of powers if the same authority still writes the rules, builds the machinery that executes them, controls the infrastructure through which they operate, and retains the ability to redesign the system the court is judging.

The architects of constitutional democracy understood this problem in a different context. Their insight was not that judges were uniquely wise, or legislators uniquely virtuous. It was that concentrated power is dangerous even when exercised by competent or well-intentioned people.

Legislative power makes rules and should be separated from the executive power that acts on them which in turn should be separated from the judicial power that judges their application. That’s not just oversight. That’s what makes power encounter power before it becomes final.

On most modern algorithmic systems, someone determines what counts as hate speech, misinformation, acceptable nudity, dangerous organizations, political content, or spam. Increasingly, these choices are translated into classifiers, ranking systems, thresholds, recommender systems, and model instructions. That is a legislative-like function. It establishes the conditions under which behavior becomes permissible, visible, rewarded, reduced, or forbidden.

The system then applies those rules at an extraordinary scale. It ranks, recommends, demotes, labels, removes, suspends, or amplifies. That is an executive-like function. Rules become action.

Finally, someone decides whether the system acted correctly. Appeals are reviewed. Metrics are selected. Errors are classified. Harms are evaluated. Policies are interpreted. Systems are retrained or left untouched. That is a judicial-like function.

These three functions ultimately return to the same organization and have quietly reconstructed the concentration that constitutionalism was designed to prevent.

What would appear unacceptable in other domains of consequential authority has become an ordinary feature of technology product development.

Oversight constrains the exercise of power. Separation changes who possesses the different powers in the first place. An external authority may inspect a system, impose requirements, certify it, or even prevent its deployment. Those can be powerful constraints. But if the builder still defines the operative rules, translates them into the system, executes them at scale, judges their failures, and determines how they should be revised, the underlying powers have not been separated. They have been subjected to oversight.

Facebook may have accidentally demonstrated the central problem that oversight fails to account for and why Dario Amodei’s proposals for AI governance do not either. It showed what happens when judicial-like power is “oversighted” but not separated, while rule-making and execution remain concentrated.

Imagine Facebook had adopted every Amodei-style regime before launching a new recommendation system for teenagers. External evaluators are embedded inside Meta. They have access to internal data and models. Meta must run safety evaluations before deployment. The system reaches a capability checkpoint and must demonstrate that it does not systematically recommend self-harm, eating-disorder, or other harmful material. Independent evaluators test it, identify several weaknesses, Meta corrects them, and eventually the system passes certification.

But Meta still decides what the system is fundamentally instructed to optimize. Now, suppose its operative objective remains something like maximizing predicted engagement subject to certain safety constraints. Meta defines the thresholds that translate abstract concepts such as “harm,” “healthy engagement,” or “age-appropriate content” into engineering specifications. Its engineers choose the ranking architecture, training data, reward signals, intervention thresholds, and trade-offs between engagement and safety.

The external evaluator can verify whether Meta complied with the certification regime. A regulator can even stop deployment if the specified requirements are violated. Yet something different begins happening after deployment.

No individual content recommendation qualifies as prohibited. No single interaction breaches the certified safety threshold. But the recommendation system learns that a certain teenager repeatedly engages with appearance-related content. Over weeks, it gradually narrows her informational environment: fitness videos become dieting content; dieting becomes extreme calorie restriction; increasingly provocative material generates slightly more engagement and therefore receives slightly more ranking weight.

Every individual step can remain within the rules. Every audit can show that prohibited content is being removed at the required rate. Every certification can remain technically valid.

And nevertheless, the system as a whole can produce the very trajectory that becomes harmful.

Who decides that this cumulative trajectory constitutes a failure requiring a fundamental redesign? Meta still does. Who decides whether the engagement objective itself must change? Meta still does. Who rewrites the recommendation rules? Meta. Who implements the new architecture? Meta. Who studies whether the changes worked and decides whether another redesign is warranted? Again, largely Meta.

Amodei’s checkpoints constrain how algorithmic power can be concentrated, but they do not separate it. They give an external actor authority to constrain the chain; they do not necessarily divide authority within the chain. They therefore remain within the category of oversight.

Oversight can become extraordinarily powerful, going even much further than observation. It can include investigation, embedded evaluators, mandatory testing and capability checkpoints, conditions on deployment, binding regulation, and even the power to stop or pause development. At its strongest, it can become genuine counterpower.

But even then, the underlying concentration can remain intact. The builder may still write the operative rules, encode them, execute them at scale, interpret their failures, and decide how the system itself should change.

It is time to learn from what did not work, especially when we already possess empirical evidence that leaves little room for speculation, rather than wait for AI to reproduce, at scale, the dramatic consequences we now know social media can generate simply because we have placed it in a different category and called it a chatbot. The category changes. The concentration of power does not.

And there is no need to invoke the extinction scenarios advanced by some to recognize the impact algorithmic systems already have today, positive in some respects, certainly, but also harmful and, unfortunately, in certain circumstances, even contributing to fatal outcomes for many .

As algorithmic systems acquire the ability to allocate opportunities, regulate speech, price risk, rank workers, mediate knowledge, and shape decisions at societal scale, some technical choices acquire the character of public power.

At that point, asking whether an algorithm is accurate, unbiased, explainable, or safe is the tip of the iceberg.

We must also ask what makes algorithmic power legitimate?

As long as frontier AI builders are capable of coding the rules, executing what they design, and adjudicating which decisions and reforms are worth implementing over billions of people worldwide, all on their own, with nothing preventing this concentration of legislative-like, executive-like, and judicial-like algorithmic power, and nothing forcing it to encounter counterpower, oversight is overridden by design.

The alternative is Trias Algorithmica 1 : the algorithmic separation of powers.

We used to say, “follow the money.” When it comes to algorithmic systems, that may be a distraction. Look instead at where authority remains, even when action is frozen. Oversight at its best asks who should be able to stop the builder. Trias Algorithmica 1 asks why the builder should possess all three powers in the first place. Trias Algorithmica 1 begins where code can’t rule. Even when every verification and certification has been passed.

That is the difference between oversight and separation of powers.

This is why the next generation of AI governance cannot stop at oversight, even on steroids.

Facebook built something that looked like a Supreme Court. That was an important institutional innovation.

What Facebook did not give that institution was the power to separate Facebook’s own powers in order to constrain the algorithmic system built around them.

Nor should we expect frontier AI builders, operating under competitive pressure, to voluntarily dismantle this concentration of authority.

Algorithmic power will become legitimate only when rule-making, execution, and judgment are arranged so that no single actor can complete the entire chain of authority by itself.

That is the premise of Trias Algorithmica 1 : separation matters, but it becomes constitutional only when the separated powers can actually constrain one another so that algorithmic power encounters power it does not own.

1 Learn more in, Trias Algorithmica: What Code Rules.