The Human Boundary

What Remains Human When Machines Become Intelligent?

The question this essay addresses is usually asked in a form that already contains its own worst answer. Will machines replace us. The form presupposes that human worth is a quantity of capacity, that the quantity is being matched or exceeded task by task, and that the outcome is therefore a subtraction problem: whatever capacities machines acquire, humans are left with correspondingly less. Everything about this framing should be resisted, and not primarily because it produces alarm rather than analysis. It should be resisted because it silently accepts a definition of humanity that a closer look shows was never adequate, independent of what any machine can now do.

The first essay in this series traced how “intelligence” has functioned historically as a residual category, redefined downward each time a machine mastered a task previously thought to require it. The same pattern, applied to personhood rather than to a single cognitive label, has a longer and more consequential history, and it did not begin with computers.

A boundary with a history of exclusion

Aristotle’s Politics, in its first book, argues that some human beings are slaves by nature, lacking the deliberative part of the soul in a functioning degree, fit to be ruled rather than to rule. The argument grounds a category of diminished personhood in an alleged deficit of a specific cognitive capacity, reason exercised in its complete and authoritative form. Two millennia later, René Descartes drew a comparable line between humans and animals in the fifth part of the 1637 Discourse on the Method, arguing that no animal could arrange words to declare its thoughts to others as even the dullest human being could, and that this incapacity for language, rather than any deficiency of bodily organs, showed that animals possessed no reason at all. Descartes’ criterion was explicitly a test: language use as diagnostic of an inner rational faculty, absence of the behaviour as proof of the faculty’s absence.

Both arguments share a structure that recurs throughout the subsequent history of the boundary. A capacity is identified, one group is judged to lack it or to possess it only in diminished form, and the judgment is used to justify differential treatment, up to and including denial of full moral standing. The identical structure was applied, across the eighteenth and nineteenth centuries and in some jurisdictions well into the twentieth, to enslaved people, to colonised populations, and to women, in each case through claims about deficient rationality, incomplete self-governance, or diminished capacity for abstract thought, claims advanced by the same philosophical culture that was, at the same moment, elaborating its most sophisticated theories of universal human dignity.

This history is not offered here as a digression from the question of artificial intelligence. It is the reason the question needs to be reframed before it can be usefully answered. If humanity has repeatedly been defined by reference to a capacity, and if every capacity-based definition on record has functioned, sooner or later, as an instrument for excluding some humans from full standing while including all humans who happened to resemble the definer, then a capacity-based definition is not a neutral starting point that machines now threaten. It is an unstable structure that was already producing the wrong answer before any machine existed to complicate it. The arrival of systems that perform reasoning, language use and creative combination at a competitive level does not introduce this instability. It exposes it, by removing the last empirical prop, the assumption that only humans could possibly satisfy the criterion, that had let the instability go unnoticed.

What the hard problem does and does not show

The most direct response to this history is to relocate the boundary from external capacities, which are visibly transferable, to subjective experience, which is not observed even in the human case and might therefore seem safe from erosion by demonstration.

David Chalmers’s 1995 paper “Facing Up to the Problem of Consciousness” gave this move its standard vocabulary. The easy problems of consciousness, on Chalmers’s division, are the ones concerning the explanation of cognitive functions: how a system discriminates stimuli, integrates information, reports on its internal states, controls behaviour. These are easy only by contrast with the hard problem, which is the question of why any of this functioning is accompanied by subjective experience at all, why there is something it is like to be a system performing these functions rather than the functions simply occurring in the dark. Thomas Nagel’s 1974 paper “What Is It Like to Be a Bat?” had already argued that this subjective character resists capture by any objective, third-person description, however complete: knowing everything about bat neurophysiology and echolocation would still not tell an investigator what the experience is like from the bat’s own point of view, because the very feature in question, the perspectival character of experience, is precisely what objective description leaves out by being objective.

This is a genuine boundary, and it does not fall to a demonstration of competent behaviour, because it was never a claim about behaviour. A system could match or exceed human performance on every observable task while the question of whether there is something it is like to be that system remained completely open, unaffected by the performance either way. This is worth stating plainly because it is routinely missed in both directions: fluent output is taken by some as evidence of inner experience, and its absence is taken by others as evidence against it, when Nagel’s argument implies that behavioural evidence is simply the wrong kind of evidence for either conclusion.

What this boundary does not supply is a criterion that reliably separates humans from machines, because it does not reliably let us confirm the boundary even among humans. The problem of other minds, discussed extensively in the empiricist tradition and given a canonical statement in John Stuart Mill’s argument from analogy in his 1865 examination of Hamilton’s philosophy, notes that no person has direct access to another person’s subjective experience. Confidence that other humans are conscious rests on an inference from behavioural and physiological similarity to oneself, not on observation of their experience directly. The inference is powerful because the similarity is extensive: shared neural architecture, shared evolutionary history, shared developmental trajectory, a body that flinches, ages and dies in the same way one’s own does. None of this is available for a system built from silicon on an engineering timeline of years rather than an evolutionary timeline of hundreds of millions. The honest position is not that machines have been shown to lack subjective experience. It is that the analogical grounds available for extending the inference, weak as they already are between humans, are far weaker again in the case of an artefact that shares almost none of the properties the human-to-human inference relies on. Uncertainty in both directions should be stated as uncertainty rather than resolved by assertion, and the asymmetry in the strength of the available evidence is itself the most defensible thing that can currently be said.

Identity was never a fixed possession

A different attempt to secure the boundary appeals to personal identity, the sense in which a person remains the same entity across time, as something intrinsically and exclusively human.

John Locke, in the chapter “Of Identity and Diversity” added to the second edition of his Essay Concerning Human Understanding in 1694, had already argued that personal identity does not consist in sameness of an immaterial soul or sameness of biological substance, but in continuity of consciousness, specifically the capacity to extend memory backward to a past action or thought and recognise it as one’s own. On Locke’s account, identity was never a metaphysical given attached automatically to a body. It was constituted by a psychological relation, one that could in principle be interrupted, fail to obtain, or be instantiated in more than one way.

Derek Parfit’s Reasons and Persons, published in 1984, pressed this line of reasoning to a conclusion Locke did not draw. Through a series of thought experiments involving teletransportation and the surgical division of a brain into two functioning hemispheres each transplanted into a different body, Parfit argued that personal identity, understood as a fact of the matter with a determinate yes-or-no answer, is not what matters in survival. What matters is psychological continuity and connectedness, relations that come in degrees, that can hold to a greater or lesser extent, and that can in principle branch into more than one successor without there being a further fact about which successor is really the original person. Parfit’s conclusion was deliberately unsettling to ordinary intuition: identity, he argued, is less deep than we tend to assume, and much of what we care about in survival can be secured by continuity relations that do not require a single, unbranching, metaphysically privileged self persisting underneath them.

The relevance of this argument to the present question is structural rather than a claim that any existing system has psychological continuity in Parfit’s sense. It is that the strongest available philosophical account of what personal identity actually consists in was already, before any AI system existed, a relational and gradational one rather than an appeal to an indivisible essence unique to biological humans. If identity is continuity and connectedness rather than possession of a metaphysical core, then the question of whether an artificial system could possess something analogous is not settled by definitional fiat in humanity’s favour. It has to be examined on the same terms Parfit used for the human case, terms that were designed from the outset to accommodate difficult and unclear instances rather than to guarantee a clean boundary.

Responsibility as a practice, not a property

The most productive reframing available concerns not what a system is, internally, but what a community does when it treats something as a responsible agent.

P.F. Strawson’s 1962 lecture “Freedom and Resentment” argued that moral responsibility should be understood through the reactive attitudes: resentment, gratitude, indignation, forgiveness, the responses we have toward one another as participants in relationships of mutual demand and concern. On Strawson’s account, holding someone responsible is not primarily the application of a metaphysical verdict about whether they possessed libertarian free will at the moment of action. It is the natural human practice of reacting to others as fellow participants in a shared moral life, a practice that would not be dissolved by a philosophical proof that determinism is true, because the practice does not rest on that metaphysical question in the first place. We suspend the reactive attitudes toward people we judge not to be appropriate targets of them, the very young, the severely impaired, the coerced, and this suspension is itself a social and practical judgment about participation in a shared practice, not a metaphysical discovery about an inner faculty.

Harry Frankfurt’s 1971 paper “Freedom of the Will and the Concept of a Person” supplies a complementary criterion, locating personhood in a structural feature available in principle to any system with the right internal organisation. A person, for Frankfurt, is distinguished from what he calls a wanton by the capacity to form second-order desires, desires about which of one’s own first-order desires should move one to act. A wanton acts on whatever desire is strongest at the moment, with no evaluative stance toward its own motivational structure. A person can want to want something different from what it currently wants, can identify with some of its own desires and disown others, and it is this reflective structure, not the mere presence of desire or the mere capacity to act, that Frankfurt takes to constitute being a person in the morally relevant sense.

Taken together, these two accounts relocate the question this essay is asking. Responsibility and personhood, on Strawson’s and Frankfurt’s analyses, are not detected by inspecting a system’s internal constitution for the presence of a metaphysical property. Responsibility is constituted by inclusion in a practice of mutual address, and personhood is constituted by a specific structural relation a system bears to its own motivational states. Both are, in principle, gradable, contestable, and open to extension or restriction by the community that maintains the practice, exactly as the boundary of who counted as a full participant in that practice has in fact been extended and restricted, wrongly in both directions, throughout the history traced earlier in this essay.

What is left when the list runs out

The pattern across every attempt examined here is the same. Reason, language, tool use, creative combination, all were at some point treated as the defining and exclusive property of human beings, and all have since been shown either to be present in non-human systems, biological or artificial, in some functioning degree, or to have been unreliable markers even among humans, deployed selectively to exclude some humans from full standing. A definition of humanity built as a list of exclusive capacities was never going to survive, because every item added to the list was added by observing what, at that moment, nothing else appeared to do, which made the list a record of current ignorance rather than a stable inventory of an essence.

What does not appear on this list, and what the philosophical accounts examined here converge on despite arriving from unconnected directions, is a specific structural feature: the capacity to be positioned within, and to sustain, relations of mutual recognition, reactive response, and continuing narrative accountability with other participants over time. This is not a capacity a system has privately and that could then be measured and matched. It is a status conferred and maintained relationally, closer in form to Wittgenstein’s account of meaning as constituted by participation in a shared practice, examined in the third essay of this series, than to a property discovered by inspecting an isolated system in the way a benchmark inspects a model’s outputs.

This reframing does not resolve the practical questions the presence of increasingly capable artificial systems will continue to raise. It changes what those questions actually are. The question is no longer whether a system has, hidden inside it, the same inner spark that makes humans count, a question the argument from other minds shows we could not answer even about other humans with certainty. The question is whether, and on what terms, a system should be admitted into the practices of mutual address, responsibility attribution and continuing accountability that have always been the actual site where personhood was decided, in the case of every group whose standing was historically contested, humans included. That is a political and institutional question as much as a philosophical one, and unlike the question of hidden inner experience, it is one a community can actually decide, has always in fact been deciding, and will go on deciding regardless of whether the deeper metaphysical question is ever settled.