In the spring of 2026, an automated algorithmic architecture nearly ignited an international military conflict between the United States and China, which was simultaneously embroiled in a war with Iran. According to a CNN investigation, a special operations analyst engaged with a chatbot to gather intelligence regarding the cargo manifest of a Chinese vessel navigating the Middle East. This system synthesized open-source intelligence with classified signals intelligence. By interpreting this data to establish what qualified as evidence of a nuclear threat, the algorithm essentially executed a quasi-normative function akin to legislative power, dictating what counted as proof of danger.

The system concluded that the ship carried components tied to a nuclear weapons initiative. The analyst subsequently utilized the platform to draft an intelligence report based on these algorithmic findings. Consequently, the assessment gained quasi-executive authority, transforming its conclusion into an actionable directive capable of setting off a cascade of military maneuvers. Distributed across the U.S. military hierarchy, the report catalyzed operational preparations to the point where aircraft were actively airborne.

By assessing its own internal conformity and validating its conclusions against the very rules it had independently interpreted—acting as its own sole proof of validity—the system effectively exercised a quasi-judicial function.

Independent oversight was bypassed entirely as the algorithm’s outputs attained institutional validity before facing external scrutiny. Lacking such checks, the conclusions were implicitly treated as self-validating, allowing the framework to default into a quasi-judicial role. American officials ultimately recognized that the system had generated a flawed conclusion, aborting the impending military strike.

In this alarming instance, much like numerous other scenarios involving algorithmically mediated choices with profound stakes, quasi-legislative and quasi-executive powers converged tightly within a single system, while independent oversight arrived perilously late or was swallowed by the platform entirely.

The Concentrated Threat of Modern AI

Contemporary algorithmic platforms, particularly those powered by advanced artificial intelligence, have resurrected the exact concentration of authority that constitutional philosophy was engineered to prevent. Such an arrangement would be deemed entirely intolerable in any other realm of consequential power. Yet in the tech sector, we dismiss it as standard product development. Concentrated power must never be assumed safe simply because its underlying intentions are presumed benign. It requires only sufficient scope, high confidence, minimal rivals, and absent counterpowers to morph into absolute domination.

The technology industry has historically operated on the inverse premise: that power concentration remains acceptable so long as motives are benevolent and innovation moves fast. However, not all innovation constitutes genuine progress, and progress cannot be bought at any price. True innovation demands social legitimacy to endure. Ultimately, the relentless accumulation of compute, data, deployment, integration, and autonomy naturally overwhelms the mechanisms of restraint upon which social legitimacy relies, unless those restraints are made structural.

Thus, our core dilemma is not merely about the algorithm as a tool, but about the societal fabric we build around it: what establishes the legitimacy of algorithmic power?

The solution cannot simply be ethics, as if moral principles alone could grant legitimate authority. Nor can it be improved regulation, as though mere compliance could settle who wields power over whom under specific conditions. Nor can it be solved by simply adding more institutions, or slowing down AI development. Ethics articulates guidelines; regulation binds them; institutions enforce them; and slowing down buys time. Yet only constitutional design establishes the foundational conditions for legitimacy by defining the separation of powers and counterpowers.

The Architecture of Algorithmic Powers

The architecture of algorithmic governance can be understood through eight distinct structural pairings:

  • Goal-setting to machine-making
  • Data stewardship to developer control
  • Training governance to optimization
  • Deployment authorization to design ownership
  • Societal signal monitoring to engagement metrics
  • Contestability to algorithmic finality
  • Compute governance to the power to scale at will
  • Market portability standard-setting to vertical lock-in

In every pairing, the former represents a governance function, while the latter signifies the mechanism through which control over that function becomes dangerously concentrated. Separating algorithmic power requires unbundling these functions to prevent their monopoly under a single locus of authority.

1. Goal-Setting vs. Machine-Making

The first separation prevents the unilateral definition of intent. Goal-setting must remain distinct from machine-making to ensure that builders do not simultaneously define aims, tolerable harms, and success criteria. Otherwise, agenda-setting and self-approval collapse legislative intent into engineering discretion. Facebook's 2018 News Feed overhaul highlighted this danger: defining meaningful social interactions as the core objective, the company engineered algorithms to reward reshares and comments, measuring success via internal engagement metrics. When researchers discovered this amplified divisive content, fixes stalled because they threatened those exact metrics. This culminated in Meta's 2026 settlement of up to $18 billion over allegations that its platforms promoted compulsive use among minors while minimizing risks.

2. Data Stewardship vs. Developer Control

The second separation ensures no single actor monopolizes the data fueling AI systems. Data stewardship must be decoupled from developer control to limit model builders' power to unilaterally collect, classify, and retain information. OpenAI's confrontation with Italy’s data protection authority in March 2023—which temporarily restricted ChatGPT over inadequate transparency and legal bases for large-scale data harvesting—illustrates why the authority to dictate training data cannot rest solely with the creators.

3. Training Governance vs. Optimization

The third separation confines optimization within society's normative boundaries. Training governance must be separated from optimization to prevent technical objectives from morphing into de facto rules. Google's 2024 Gemini image-generation controversy demonstrated this risk; tuning the system for diversity failed to account for historical contexts, leading to inaccurate depictions and over-cautious refusals. Independent training governance establishes these vital boundaries before optimization dictates model behavior.

4. Deployment Authorization vs. Design Ownership

The fourth separation conditions AI societal entry on independent release authorization. Deployment authorization must be separated from design ownership to restrict creators from unilaterally deciding when and where a system enters daily life. Cruise's autonomous-vehicle rollout in California exemplifies this; despite owning the self-driving technology, Cruise faced permit suspensions from the California DMV in late 2023 over safety concerns, proving developers cannot be the sole arbiters of public road safety.

5. Societal Signal Monitoring vs. Engagement Metrics

The fifth separation places societal impact evaluation beyond internal corporate discretion, ensuring independent oversight remains embedded throughout the active deployment lifecycle.