The Architecture of Intelligence

How Information Becomes Understanding

The standard picture of a cognitive system is a sequence. Something arrives at the sensory surface, perception converts it into a usable form, memory stores it, representation encodes it, inference operates on it, and understanding appears at the end as the output of the chain. The picture is intuitive and it organises most public discussion of what machines can and cannot do. It also fails at every junction, and the way it fails determines what can be asked about artificial systems.

The picture has a traceable origin. Claude Shannon’s 1948 model of communication established the vocabulary of source, channel, encoding and receiver, and its success in engineering made it available as a general metaphor for cognition. Donald Broadbent’s Perception and Communication, published in 1958, imported that vocabulary into psychology and proposed attention as a filter positioned between sensory input and processing capacity. David Marr’s Vision, published posthumously in 1982, gave the picture its most disciplined form by separating the computational, algorithmic and implementational levels of description, which allowed a cognitive function to be specified independently of the machinery that realised it. Jerry Fodor’s The Modularity of Mind, in 1983, added the claim that input systems are informationally encapsulated, operating on their own proprietary data without access to what the rest of the system knows.

Each of these was a productive simplification. The trouble is that the components they separated turn out to presuppose one another, which means the sequence has no first term.

The conditions do not stack

Perception is described in the pipeline as transduction, the conversion of physical energy into information. Hermann von Helmholtz identified the difficulty with this description in 1867, in the third volume of the Handbuch der physiologischen Optik, where he argued that visual experience is the product of unconscious inference. The retinal image is systematically ambiguous. A given pattern of light is compatible with an unbounded set of scene configurations, and the visual system resolves the ambiguity by applying assumptions about how the world is typically arranged. Those assumptions are prior knowledge in a functional sense. Perception therefore operates on representations before it delivers them, and cannot occupy the first position in the chain.

Memory is described as storage and retrieval. Frederic Bartlett’s Remembering, published in 1932, established that recall is reconstructive: participants asked to reproduce an unfamiliar narrative progressively normalised it toward the conventions of their own culture, dropping incongruent elements and inventing connective material that had never been present. Elizabeth Loftus and John Palmer demonstrated in 1974 that the phrasing of a question asked after the fact alters what is subsequently reported as remembered, with speed estimates for a filmed collision shifting according to the verb used in the interrogation. Memory is therefore modified by inference and by language, both of which the pipeline places downstream of it.

Memory is also not one system. Endel Tulving’s 1972 distinction between episodic and semantic memory separated the recollection of particular events from the retention of general facts. The patient Henry Molaison, following bilateral medial temporal resection in 1953, lost the capacity to form new declarative memories while retaining the capacity to acquire new motor skills, improving at mirror-tracing tasks across sessions he had no recollection of performing. A single word covers dissociable mechanisms with different anatomies, and the same is true of every other term in the list.

Representation is described as a neutral container. It is closer to a set of constraints on what can subsequently be computed. A propositional format makes logical relations explicit and spatial relations expensive. A metric format inverts that. John McCarthy and Patrick Hayes identified the frame problem in 1969 by showing that a system representing a world as a set of propositions must somehow determine which propositions remain unchanged after an action, and that stating the invariances explicitly produces a combinatorial explosion. The choice of representation determines which questions are tractable, which means that representation is a substantive commitment rather than a transparent medium. Stevan Harnad added the complementary difficulty in 1990: symbols acquire their content from their relations to other symbols, and a system in which every symbol is defined by other symbols has no point of contact with anything outside itself.

Language and thought come apart

Language is the condition that most reliably gets conflated with intelligence, for a straightforward reason. It is the only cognitive capacity whose products are directly observable. Everything else is inferred from behaviour, while language presents itself as behaviour and content simultaneously.

Its contribution to cognition is real and specific. Compositional structure allows a finite vocabulary to express an unbounded set of propositions, which permits reasoning about situations never encountered. It permits the transmission of knowledge acquired by others, which removes the requirement that each learner discover everything individually. It permits the fixing of abstractions that have no perceptual correlate, which is the precondition for mathematics and law.

The neuropsychological evidence nonetheless shows that language and reasoning are separable. Patients with severe agrammatic aphasia, studied by Rosemary Varley and colleagues from the late 1990s, have been shown to retain arithmetic competence, causal reasoning and theory of mind, performing operations whose structure their linguistic system can no longer express. Work in Evelina Fedorenko’s laboratory has mapped a language-selective network in the left frontal and temporal cortex that shows little response during arithmetic, logic and working memory tasks, and that is distinct from the domain-general network engaged by demanding reasoning across content types. Language is one condition among several, and it can be removed while others persist.

The reverse dissociation is the one now visible in artificial systems. A model trained on next-token prediction acquires very high linguistic competence, including syntactic well-formedness, register control and discourse coherence, while exhibiting failures of inference that no competent speaker would produce. The two capacities that we have never observed apart in a human being have been observed apart in an artefact, and this is the central empirical fact about the current situation.

The problem of selection

Inference is described as the application of rules to premises. The description accounts for deduction, where the conclusion is contained in the premises and the operation is truth-preserving. It accounts poorly for the other two forms.

Induction, the extension of a regularity beyond the observed cases, has no formal justification. David Hume established in 1739 that any argument from past regularity to future regularity presupposes the uniformity of nature, and that the uniformity of nature can be established only by an argument from past regularity. The circle has never been broken, and inductive practice proceeds without it. Abduction, the inference to the best available explanation, was identified by Charles Sanders Peirce as the only form that introduces new content. It has no algorithm, because generating candidate explanations requires already knowing which features of a situation call for explanation.

That last requirement is the deeper problem, and it is not confined to abduction. Any system with a non-trivial knowledge base can draw an unbounded number of valid inferences at any moment, almost all of which are irrelevant. Reasoning consists largely in not drawing them. The operation that determines which inferences are worth performing is not itself an inference, since evaluating candidates would require generating them first. This is the frame problem in its general form, and it is the reason that formal accounts of reasoning have consistently underdescribed the phenomenon they model.

Context is the condition that performs the selection. Its role is easiest to see in interpretation. Paul Grice showed in 1975 that the content communicated by an utterance regularly exceeds what its words encode, and that hearers recover the excess by assuming the speaker is being cooperative. Dan Sperber and Deirdre Wilson argued in 1986 that the recovery is governed by a trade-off between the cognitive effect an interpretation yields and the effort it costs, with processing halting at the first interpretation that satisfies the balance. On either account, the same signal carries different content in different contexts, and no fixed mapping from input to content exists. Context is not additional information supplied alongside the message. It is the condition under which the message has content at all.

Learning is not a separate stage

Learning is described as the acquisition of content by a system whose architecture remains fixed. What learning actually modifies is the other conditions: which distinctions perception makes, which representations are available, which inferences are cheap, which contexts are recognised.

Two failure modes make the dependency visible. Michael McCloskey and Neal Cohen described catastrophic interference in 1989, showing that a connectionist network trained on a new task can lose performance on a previously learned one, because the weights encoding both are the same weights. Distribution shift describes the collapse of learned performance when deployment conditions differ from training conditions in ways the system has no means of detecting. Both are consequences of the same structural fact: a learning system has no independent access to the criterion by which its own generalisations should be constrained. It has a loss function, and a loss function specifies what counts as an error only relative to a fixed distribution.

The requirement that learning presupposes a criterion of error, and that the criterion presupposes a representation, closes the circle begun with perception. There is no layer at which raw material enters and no layer at which interpretation is added. Every condition operates on material already shaped by the others.

Four terms

The four concepts that organise this inquiry can now be given working definitions, stated as functional relations rather than as levels of a hierarchy.

Information is a difference that constrains the set of states a system can be in. It is defined relative to a receiver with a prior state space, which is why the same physical event carries different information for differently constituted systems and none at all for a system with no relevant states.

Knowledge is information whose reliability is underwritten by something outside the information itself, whether a causal connection to what it concerns, a method whose error rate is known, or an institution that certifies it. The underwriting is what distinguishes knowledge from correct guessing, and it is external to the content, which is why no examination of a claim in isolation determines whether it constitutes knowledge.

Reasoning is any transformation of representations that preserves a specified property. Deduction preserves truth. Statistical inference preserves calibration. Analogy preserves structural relations while permitting content to vary. Naming which property is preserved is the substantive part of the description, and accounts of reasoning that omit it describe nothing in particular.

Understanding is the capacity to redeploy a representation under counterfactual variation. A system understands a domain to the extent that it can answer questions about what would happen if conditions differed from those it has encountered. Judea Pearl’s hierarchy, set out with Dana Mackenzie in 2018, gives the criterion a formal shape: association answers what is observed, intervention answers what follows from acting, and counterfactuals answer what would have followed under conditions that did not obtain. Each level requires assumptions the level below cannot supply, and no volume of observational data lifts a system from one to the next.

This criterion has a property the behavioural tests lack. It specifies where to look for the difference between a system that has understood and one that has matched a pattern, namely at performance under conditions systematically absent from training. It does not resolve the constitutive question raised in the first essay, since a system could satisfy it and still lack whatever Searle was pointing at. It does make the descriptive question tractable.

What follows for evaluation

If cognitive competence consists of separable conditions that constrain one another, then a single scalar measure of intelligence carries almost no information about a system. Two systems with identical benchmark scores can differ completely in which conditions they satisfy, and the difference determines how each will fail.

The practice of evaluating artificial systems by aggregate performance on task batteries inherits its form from psychometrics, where the aggregate is defensible because the components correlate strongly across human populations. That correlation is a fact about human beings, produced by shared developmental architecture and shared exposure. It is not a fact about cognitive systems in general, and there is no reason to expect it in artefacts built by a different route. Applying the human aggregation to machines assumes the conclusion that the machines are organised as we are.

The alternative is to evaluate each condition separately and to report the profile rather than the sum. Which distinctions does the system’s perception support and which does it collapse. What does its representational format make expensive. Does its linguistic competence exceed its inferential competence, and by how much. What determines its selection of relevant material. How does its performance change when the deployment distribution shifts. These are answerable questions, and the answers differ across systems that current benchmarks rank as equivalent.

The object of this inquiry

The systems under discussion will be replaced. Architectures current in 2026 will be obsolete on a horizon of a few years, and analysis anchored to their specific properties expires with them. The conditions are more durable. Perception, memory, representation, language, inference, learning and context are constraints on any system that must act on incomplete information in a world it does not control, and they will constrain the successors of current systems as they constrained their predecessors.

This project examines those conditions and the concepts built on them. It treats particular systems as evidence about the conditions rather than as subjects in their own right, and it treats claims made about those systems as objects of analysis, including claims about what the systems understand. The next essays take the four terms defined above and examine each against specific cases.

The Machine Question is an independent research project by Paolo Calvi.