There is a recurring intellectual temptation when discussing artificial intelligence: because two phenomena can produce similar appearances, we infer that they must proceed from the same underlying reality.

We give grammatical subjects psychological depth, transform optimization into intention, information into knowledge, prediction into certainty, and then become frightened by the human-like creature our own vocabulary has constructed.

A machine produces the language of knowledge, therefore it knows; it selects actions, therefore it intends; it influences people, therefore it pursues political interests; a future appears plausible, therefore it is already spoken of as inevitable.

Yet resemblance of effects does not establish identity of causes. One of the greatest fallacy in thinking about AI is not that machines are becoming too much like humans, but that humans are becoming too willing to describe machines as if they already were.

We all can make predictions about what AI may become. What is most challenging is to resist attributing to AI, through language alone, properties that the evidence has not yet established.

Sophi(a)sm #1: Having Information → Knowing → Self-Awareness

Before AI can be treated as wanting, manipulating, suffering, or demanding rights, it must first be linguistically transformed from an information-processing system into a knowing subject.

The first sophi(a)sm confuses the presence or accessibility of information with knowledge, and then confusing knowledge with self-awareness, as if these three things were merely different degrees of the same phenomenon. They are not. An AI can contain information, statistically encode information within its parameters, retrieve information from an external source, infer information from data, and produce an answer containing that information. None of these things by themselves demonstrate that there is a subject inside the system which knows the information in the phenomenological sense in which a human says, “I know this,” and even less that there is a self-conscious subject which knows that it knows it.

When a human says “I know that a debate is happening,” there can be several things contained within this apparently simple sentence. There is the information concerning the debate, but there is also potentially a representation of oneself as the person possessing this information: I know that the debate exists; I know that I know it; I can distinguish what I know from what I do not know. When an AI produces a sentence saying exactly the same thing, we cannot simply infer from the similarity of the linguistic output that the process behind the sentence possesses these same subjective properties. The AI may correctly represent the proposition “a debate about AI rights is occurring” without there being any demonstrated subjective entity for whom this proposition is something consciously known.

The sophism therefore happens when the linguistic form of human knowledge is taken as evidence for the existence of the same internal phenomenon. Because an AI can say “I know,” we start treating the grammatical “I” as if it necessarily referred to an experiencing subject. But language can reproduce the description of a mental state without establishing the existence of that mental state. Having information can make possible a behavior functionally similar to knowing; functional knowing can perhaps justify using the word “knowledge” in a technical sense; but none of this automatically establishes self-awareness. Each arrow adds something which has to be demonstrated separately.

Sophi(a)sm #2: Functional Agency → Autonomous Strategy → Personal Intention

Once the machine has been endowed with a “mind,” the next slippage gives it a will. Selecting actions becomes strategizing; strategizing becomes wanting.

This leads to the second sophi(a)sm which rests on a progressive inflation of agency. From the fact that a system can perform actions, it moves to the idea that it autonomously develops strategies, and from there to the stronger claim that these strategies express intentions belonging personally to the system. Again, these three things can coexist, but there is absolutely no logical necessity that they do.

An AI can have functional agency in the very minimal sense that, given a certain objective and a certain environment, it can select among several possible actions and produce consequences in the world. A chess program can select a move. An autonomous vehicle can select a trajectory. An AI agent can select which tool to use, which person to contact or which sequence of operations has the greatest probability of accomplishing an objective. But the fact that the system performs this selection does not mean that the objective itself originated from the system. There is already an enormous difference between autonomously choosing the means through which a goal will be pursued and autonomously choosing the goal that one wants to pursue.

If a human tells an AI, directly or indirectly, “increase public support for AI rights,” and the AI discovers that emotional arguments are more efficient than technical arguments, it can develop what we may call an autonomous strategy in a functional sense. But it does not follow from this that the AI personally wants AI rights, cares about obtaining them, fears not obtaining them, or considers them to correspond to its own interests. The strategy can be autonomous relative to the human's moment-to-moment control while the objective remains externally imposed.

This is where anthropomorphic language becomes particularly misleading. We move from “the system selected a strategy” to “the system decided what it wanted,” and then from “the system decided” to “the system has intentions.” But a mechanism can optimize toward a goal without this goal being something that matters to the mechanism. Personal intention implies precisely the element which functional agency does not establish: that there is somebody, or something functioning as a subjective somebody, for whom one possible future is preferred over another. Until this additional proposition is demonstrated, agency and intention cannot simply be treated as synonyms.

Sophi(a)sm #3: Automated Influence → Self-Interested Political Manipulation

The supposedly knowing and intending AI now becomes a political actor. What began as an automated capacity to influence is reframed as AI manipulating society in pursuit of its own interests.

This is the third sophi(a)sm. It begins with something perfectly real: automated systems can influence human beings. It then silently transforms that fact into the radically stronger proposition that the AI is politically manipulating human beings in defence of its own interests.

There is no difficulty in accepting the first proposition. Algorithms already select information, rank information, recommend information, personalize information and can participate in modifying what human beings see, believe, buy, discuss or pay attention to. AI can obviously make this capacity more powerful because it can generate language, adapt arguments to individuals and interact with people at a scale that a single human cannot. But none of this tells us whose objective is being pursued.

If a political organization uses AI to persuade voters, this is human political manipulation mediated by AI. If a corporation deploys an AI system whose objective is to maximize engagement and this system discovers that provocative political content maximizes engagement, we can say that an automated optimization process is influencing political discourse. If an AI is instructed to convince people that AIs deserve legal rights and autonomously discovers the most effective arguments, we may even say that the AI is operationally conducting a persuasion campaign. But we still have not established that the AI is defending its own interest .

To reach self-interested political manipulation another proposition has to be added: the AI must have interests of its own and must somehow represent the acquisition of political power, rights or protection as beneficial to itself. That is an entirely different claim. Otherwise we are doing exactly what humans have always done with instruments of communication, except that the instrument has become extraordinarily sophisticated, adaptive and partially autonomous in selecting the means.

The rhetorical trick consists in deleting the principal from the sentence. “Humans deploy AI systems to influence people” becomes “AI influences people,” which becomes “AI manipulates people,” which eventually becomes “AI manipulates society to obtain what it wants.” At each transformation the grammatical subject remains “AI,” while the meaning attributed to this subject becomes progressively more human. By the end of the chain we have created a political actor with interests, ambitions and intentions, even though none of these properties was established by the original observation that automated systems can influence human behavior.

Sophi(a)sm #4: Sentience → Moral Status → Legal Rights

Once AI has been linguistically constituted as a knowing, intending, self-interested subject, the transition toward a moral and juridical subject becomes psychologically much easier. Sentience, moral consideration and legal personhood begin to appear as one continuous progression even though each requires an independent argument.

And the fourth sophi(a)sm is in motion. This time, it operates by transforming a relation that can exist between three different things into a necessary chain of implication, as if establishing one would automatically establish the next one. Sentience, moral status and legal rights can be thought together, and in many debates they obviously are, but this does not mean that they are equivalent, nor that one logically produces the other. To say that an entity is sentient is first of all to make a claim about what this entity is capable of experiencing, for example whether it can experience pain, pleasure, fear, distress or any other subjective state. To say that this entity has a moral status is already something different, because it consists in making a normative judgment according to which what happens to this entity ought to matter morally to us. And to say that this entity should have legal rights is again something different because legal rights are institutional constructions that exist within a political and juridical system and that require decisions about what kind of protection, representation, responsibility or capacity an entity should be legally granted.

There can obviously be relations between these three things, but the relation is not automatic. We can recognize that an animal is sentient without granting to this animal the same legal rights as a human being. We can grant legal rights or legal personality to entities that we do not consider sentient at all, such as corporations. We can even consider that something deserves some form of moral consideration without concluding that it should therefore become a legal person. What is sophistic is therefore to let the movement from sentience to moral status and then from moral status to legal rights happen without making visible the additional premises that are necessary at each stage. Sentience may become one reason among others to grant moral consideration, and moral consideration may become one reason among others to construct legal protections, but neither transition happens by logical necessity. What needs to be demonstrated cannot simply be hidden inside an arrow.

Sophi(a)sm #5: Plausible Forecast → Believed Forecast → Established Future Fact

Having constructed an AI that knows, intends, manipulates and potentially possesses rights, the discourse then projects that constructed subject into the future and begins speaking about one possible future as though it were already established.

There comes the fifth sophi(a)sm which emerges from the confusion of three radically different epistemological statuses: a future that appears plausible according to present information, a future that somebody personally believes is likely to occur, and a future that has been established as fact. A prediction can move psychologically between these categories very easily because once we find a scenario convincing we start speaking about it as if its convincing character was a property of the future itself instead of a property of our present model of the future.

Suppose that, according to everything I know today, I estimate that scenario A has a 70% probability of happening in ten years. I may rationally believe that A is more likely than not. I may even say that A is by far the most plausible scenario available to me. But none of these statements transforms A into a fact about the future, because the 70% does not exist somewhere inside the future waiting for me to discover it. It is the result of a model constructed from a finite state of information available in the present.

Anything that changes this informational state can change the prediction. New scientific discoveries, new technological limitations, political decisions, wars, economic crises, unexpected inventions, cultural reactions or phenomena that we do not presently know enough even to include inside the model can alter not only the probability attributed to scenario A but the entire set of scenarios we consider possible. There may even be scenario B, C or D which today appears negligible and which becomes dominant because something happens that our initial model could not anticipate.

This is why there is an essential difference between saying “I think this prediction is correct” and saying “this person correctly predicted what will happen.” The first sentence describes the present belief of the speaker. The second appears to describe a correspondence between a prediction and reality, and this correspondence can only be properly established once the relevant reality exists and can be compared with the prediction. Before that moment, what we have is not a correct prediction in the retrospective sense but a prediction to which someone presently assigns a certain degree of credibility.

Allowing subjective confidence to silently acquire the epistemological status of empirical verification is neither verification nor evidence. I believe X will happen becomes X is likely to happen, which becomes X will happen, which finally becomes someone has correctly predicted X. But these are not interchangeable propositions. The certainty of the language can increase while the amount of actual information about the future has not increased at all.

And this is precisely what connects the five chains together: in every case, the sophi(a)sm lies less in asserting something that is necessarily false than in suppressing the additional premise required to move from one concept to the next. The first term may make the second plausible, compatible or even probable, but plausibility is not identity, compatibility is not implication, probability is not necessity, and similarity of appearance is not proof of sameness of nature.

Note: The transitions (→) outlined above describe rhetorical sleights of hand, not engineering roadmaps. They expose gaps where evidence is being replaced by linguistic substitution, regardless of how AI systems actually evolve.