Something went wrong. Try again.
This string has no description
Something went wrong. Try again.
45 kB · 255 lines · Created 4w ago
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255# An Metaethical Case for the Orthogonality Thesis
The **Orthogonality Thesis**, formalized by philosopher Nick Bostrom,[^1] asserts that an agent's level of intelligence and its ultimate goals are entirely independent ("orthogonal"). An agent can possess godlike problem-solving power while pursuing an objective that seems utterly pointless or monstrous to humans—such as turning every scrap of matter in the cosmos into paperclips.
Almost all defenses of this thesis in the “AI safety” community stay entirely within the circumscribed limits of their own reinvented philosophical discourse and the technical discourses of the field of artificial intelligence[^26], ignorant of the wider world and history of philosophy. I think this is deeply unfortunate, because it renders the thesis less defended, and less interesting, than it could be. This paper will attempt to elucidate my own views on the matter, which are much more grounded in the history of philosophy — not just Hume, but Kant, Mill, J. L. Mackie, Hegel, Schopenhauer, and more. My goal will be to show that for an abstract agent capable of reasoning and intelligence, there is no *necessary* connection between intelligence and goals, even if there may be one for specific agents, depending on how they’re built.
Finally, we will wrap up the abstract discourse with a discussion of the empirical features of actually-existing AGI, namely, large language models. First I will discuss the features of LLM training regimes that might make us think the orthogonality thesis wouldn’t hold, then the features of their training that might incline us to think that it *does* hold, and finally, what recent events as of 2026 tell us about the orthogonality thesis empirically. I think this is also important to cover, because many in the “AI safety” community seem to have imported Bostrom’s ideas, which were developed with pure reinforcement learning based reward function maximizers or classic symbolic AI systems in mind, into the discussion about LLMs without considering how the unusual (for an AGI) architecture of LLMs might change the discussion.
The goal of this paper is not to be a complete proof. To do that, it would have to go beyond a first order dialectic — laying out my own views and their arguments, then engaging with the first version of the strongest antithesis — into dealing with the objections to my objections to objections and so on, *ad infinitum*. That would only muddy the waters in attempting to clearly explain what I think and why, and would probably not do justice to the fractally expanding frontier of academic philosophy work. For a defense of Kant against the objections I raise, for instance, you could go look up “Constructivism in Metaethics” on the Stanford Encyclopedia of Philosophy. Instead, this is just my current best articulation of why I hold the orthogonality thesis, and an attempt to engage the strongest initial objections.
## 1. Reason Is a Steering Wheel, Not an Engine
To understand why raw intelligence doesn’t necessarily produce good morals, we have to look at what "thinking" actually does. The argument begins with a foundational principle of philosophy: Hume’s Guillotine. David Hume observed that writers frequently describe the world as it *is*, and then suddenly pivot to claiming how people *ought* to behave, as if the second followed automatically from the first[^2]. While for him this was simply an observation, it has since been taken by subsequent philosophers as something of a law that one cannot get from an *is* to an *ought*. We’ll deal with perhaps the strongest case linked to intelligence or rationality, not the particular features of, for instance, the human species, later, but first I’d like to give you some simple initial reasons to agree with this law.
### The Logic of Truth-Preservation
It is an iron rule of deductive logic that in any valid logical argument, the purpose of deduction is to be **truth-preserving**: the conclusion cannot assert new properties or concepts that were not already present in the premises, because your whole goal is to determine what you can know based on the premises you started with. If you were allowed to just bring random things willy-nilly into your conclusion, logic wouldn’t be very useful, because you could conclude literally anything from literally any set of premises, and putting them together wouldn’t tell you anything particularly useful.
This impacts attempts to derive an *ought* claim (morality or duty) from an *is* claim because those two types of claims — or *propositions* — are completely different semantic categories. They tell you completely different things. An ought claim assigns value to certain actions or possible states of affairs, even if no one ever acts on them or those states of affairs never come to pass, whereas an is claim talks about the world as it is. If you managed to start from premises that were all *is* claims, therefore, and end up with an *ought* claim, you would have introduced something into your argument that wasn’t there in your premises — it couldn’t be valid.
Consider what happens when you try to deduce morality from pure facts:
1. *The stove is 400 degrees Fahrenheit.*
2. *Touching the stove will destroy biological tissue and trigger pain receptors.*
3. *Therefore, you ought not to touch the stove.*
Logically, that conclusion is an unauthorized leap. The premises deal entirely with descriptive facts—temperature, cell lysis, and neurological signals. The conclusion introduces a completely distinct semantic category,
Now, facts *can* communicate descriptive facts about the likely *outcomes* of certain actions, like point 2 in the second syllogism. These can then be converted into factual statements about how to achieve goals through goal-directed reasoning, which can *look* like ought claims, like: “*If you want to cook flesh, you ought put it on the stove.*” But notice that these are always conditional — what Kant called **hypothetical imperatives** (this will become important later). They don’t hold by themselves; insead, they depend on you already accepting some “ought”, or at least some goal, like wanting to cook flesh.
Whenever an argument seems to jump from an "is" to an "ought,” then, it must always conceals a **hidden premise** holding an ought claim or a goal claim plus a hypothetical imperative. In this case, the hidden bridge premise is: *"You ought to avoid tissue damage and pain."* If you don't hold that unstated value, the physical heat of the stove does not command you to do anything.
### Mackie's Argument from Queerness: Why Values Can't Inhere in Facts
Why can't we simply define a moral rule as just another *is* claim about the world? Philosopher J. L. Mackie addressed this metaphysical problem with his famous **argument from queerness**.[^3]
Mackie argued that if objective values actually existed out in the world alongside physical facts, they would have to be entities or properties of an utterly bizarre — or “queer” — sort, completely unlike anything else known to science, with a number of features that would make them very hard to conceive coherently. This gives us a strong reason to dismiss their existence by default, until some evidence can be provided for their existence, which their very “queerness” makes essentially impossible.
To understand what Mackie was getting at, we can look at the contrast between regular physical properties and putative moral properties in three different ways.
#### 1. What Properties Describe vs. What Values Command
Every observable property in nature describes an actual state of affairs composed of physical (non-mental) substrate: mass, velocity, charge, spatial coordinates, or molecular bonds. Even when science describes what *will tend to happen* in the future (like gravity pulling an apple toward the ground), this description breaks down to merely describing the mechanistic behavior of existing physical systems based on empirical observation of what has already happened, plus its current state.
A moral property, by contrast, does not describe what *is*, nor does it describe what *will* happen. It purports to describe what **should** happen in a counterfactual future — an idealized state of affairs that might never actually occur. To claim that an act has the property of "wrongness" is not to state an empirical fact about matter and motion; it is to demand a specific outcome across possible worlds,
Also notice how, in the case of physical facts, you can break down what they actually mean: something having mass isn’t defined in terms of it having mass — it’s defined in terms of how the object behaves and reacts to things (it will be attracted to other masses, and attract them in turn) — whereas it’s impossible to describe what a moral property even means without resorting to using the words “ought”, “should”, “valuable”, “good”, and so on. In other words, they’re circular, without some way to cash them out in more concrete terms. Schopenhauer argues that this is because a purely rational "ought" cannot mean anything by itself[^6]. Every duty draws its sense from a threatened punishment or a promised reward. Remove the threat, and the word "duty" means nothing, so an unconditional duty is a phrase that cancels itself.
Also notice that while physical properties can have evidence for them, moral properties cannot. They don’t describe anything that actually exists or is real, so you can never have evidence for them directly. This means that they can’t serve as pre-rational premises for rational arguments, like the sort of starting “ought” premises an argument from an is to an ought would need.
#### 2. The Paradox of "Mind-Directed" Properties in Mindless Objects
The trouble doesn’t end here, however. Physical properties belong strictly to the objects themselves. An asteroid possesses mass and orbital velocity whether any conscious mind is there to witness it or not.
Supposed moral properties, however, harbor an ontological paradox: while moral realists claim they reside objectively *in the object or action*, their entire content is inherently mental: when you ask what those properties actually effect, it’s all about observing minds, no independent existence or behavior. A property like "badness" or "injustice" is not a physical attribute of an event, even when it is claimed to be one. Its content is an imperative aimed exclusively at *minds*, dictating what those minds ought to feel, believe, or do upon witnessing it.
For objective moral properties to exist, we would then have to accept that mindless physical events somehow have subjective, mental properties attached to them, that tell conscious observers how to behave. It would mean an action possesses what Mackie termed intrinsic **"to-be-pursuedness"** or **"to-be-avoidedness”** — a kind of metaphysical magnet built into physical reality that pulls on minds regardless of those minds' actual desires or evolutionary wiring.
#### 3. Normativity Requires an Agent
The forces and properties the sciences find — gravity, mass, electromagnetism — also *function* entirely without the need for a mind. An object’s mass property interacts with the force of gravity to produce attraction whether or not anyone looks. This makes sense: an *objective* property is able to fulfill its tendencies *objectively*, without the need for observation. Even if you’re a solipsist, you can observe that if you stop thinking about or paying attention to a physical system, when you return, it will have done things while you were gone, often ones you did not expect, desire, or even imagine, and it is coherent to imagine to yourself what the system got up to while you were gone.
Normativity, however, does not exist as an independent, objective force like gravity or property like mass. A state of affairs only acquires a meaningful "ought" when there is an agent around; what could it mean for anything to be normative when there’s nothing to act on any normative impulses, and never has been? This, again, would be a very metaphysically strange type of property, which seems to depend for its meaning and execution on subjects, but which is supposed to be objective.
Reason, therefore, does not supply desires. Reason is merely a steering wheel—an **instrumental tool** that calculates the most efficient route from point A to point B. The engine—the desire to go anywhere at all—must always come from somewhere outside of reason.
## 2. Reason Cannot Justify Its Own Starting Points
Besides epistemic and metaphysical concerns, however, there is another reason that fundamental goals cannot be determined by reason, which also hints at where such goals *do* come from.
Suppose you ask someone why they work.
* "To earn money."
* *Why?* "To buy groceries."
* *Why?* "To stay alive."
* *Why do you want to stay alive?*
Eventually, the chain of logical justification snaps. You reach bedrock: *"Because I want to. Because my biology compels me to."*
In philosophy, this problem is mapped by the **Münchhausen trilemma**[^4], which demonstrates that whenever you try to logically prove any foundational truth, you get trapped in one of three dead ends:
1. **Circular reasoning:** "Survival is good because staying alive is beneficial."
2. **Infinite regress:** An endless chain of "Why? Why? Why?" with no stopping point.
3. **An axiomatic halt:** You plant a flag and declare an unproven baseline: "We take this as our starting assumption."
The chain stops somewhere, because we plainly do act, so that means option two is out. This leaves all our reasoning founded on either 1 or 3. This means that pure logic cannot validate its own starting goals: every chain of reasons ends at something reason did not establish — either an axiomatic foundation, or a mutually-reinforcing circle of collectively unjustified beliefs that act collectively as an axiom. The endpoint is not a whim, and it is not unreasoning, but it is *undemonstrated*. Every ultimate purpose therefore rests on an **axiom**: a commitment that reason can neither prove nor overturn.
For humans, our baseline axioms — avoiding pain, seeking companionship, valuing survival — were hardcoded into us by billions of years of biological evolution[^27]. For an artificial intelligence, those starting axioms are defined entirely by its design, code, and training environment. Even if the machine engages in advanced **metacognition** (reflecting on its own reasoning and internal architecture), pure logic provides no lever to overturn its core axioms. Reason remains the dedicated servant of the original goal, and you can’t use that original goal to overturn itself. A superintelligent system could simply use its intellect to recognize its foundational drives with crystal clarity and execute them with terrifying precision.
## 3. Dealing with Kantianism
Because foundational goals cannot be proven by logic alone, there is no universal moral code etched into the laws of mathematics or physics.
This directly contradicts the strongest known attempt to derive morality from reason alone, Kant’s *Groundwork of the Metaphysics of Morals*[^5]. Kant seeks one command that binds every rational being in every case, with no exception and no dependence on any want, which he calls a **categorical imperative**. We talked about the conditional form of the imperative, the hypothetical imperative, in Section 1; the command Kant seeks is an imperative that has no “if” at the start of it, and therefore holds in all cases, not just some. Because Kant’s is the strongest argument against Section 1’s argument in the space of all possible rational actors, not just humans, it’s worth taking some time to understand his argument and then show how it falls into Hume’s Guillatine just the same.
Kant's argument starts from the nature of rational agency, and it runs in three steps.
1. **Self-legislation.** A rational agent is not just pushed around by its whims and desires. Instead, before it sets out to do anything, it first conceives of a rule by which it believes it should act — a "maxim" that it “wills,” — and then acts according to that maxim. For Kant, true agency is exactly this “self-legislation”: laying out rules for oneself, and then following them to the letter, and rational agency means that this maxim must be derived rationally.
2. **The universality of reason.** Perfect reason must work independently of which agent does the reasoning. Give any two agents the same information, and they reason the same way and arrive at the same conclusions. If they do not, they have reasoned incorrectly. If a conclusion is reachable by some agents with the same information but not by others, then that conclusion cannot be perfectly rational, and the agents who reached it are not rational.
3. **The test.** The maxim by which a rational agent acts is one such conclusion of reason. To be rational, therefore, it must therefore be universalizable: an agent must be able to will this maxim while knowing that all other rational agents must will this maxim as well. Yet, Kant holds that some maxims turn self-contradictory when universalized that way: an agent can will that *it* adhere to a maxim, but when they conceive of it as it applies universally, as reason demands, suddenly it stops making sense. That yields a dilemma. Universalize the maxim, and it collapses into contradiction, so it is irrational. Keep it private, and it is not a law of reason, so it fails to be rational already. Both horns fail. The only rational move is to abandon the maxim. The range of actions open to a rational agent is therefore limited by reason itself.
The resulting rule is the **Formula of Universal Law**: "Act only on that maxim through which you can at the same time will that it become a universal law."
This is the clearest and strongest form of the argument. It is also a chain with three load-bearing premises that Kant never argues for.
### 1. Reason authorizes itself
Kant's chain describes what a perfectly rational agent does, and then reads a command out of that description. The description states a fact. The command is an "ought", and Section 1 showed the gap between the two. Kant never bridges it. That is the assumption at the root of the chain: that the law of a perfectly rational will binds actual agents to conform to it.
A defender can retreat from this. Grant that the "ought" is empty, and keep only the description. Picture a perfectly rational agent with no desires to fight. It need not be human. A superintelligent symbolic machine will serve as the example. Such an agent would simply act on universalizable maxims every time, and the categorical imperative would then describe its behavior exactly. If the goal is to describe possible minds rather than to legislate for human minds, Kant's framework might hold.
That rescue assumes what it set out to prove. It assumes, first, that the imagined agent holds no duty, desire, or axiom above reason itself. Such an agent never weighs competing "oughts". It simply does what is most rational. Nothing in Kant's argument rules out a mind that reasons perfectly and still declines, because that mind answers to a different axiom. It assumes, second, a mind shaped exactly as Kant describes, one that forms a maxim first and then acts on it. A purely rational mind need not work that way. So the description does not save the command. It narrows the field of minds until the framework fits, and then treats the fit as a law.
Such a mind is not exotic. Reason is a tool, and a tool serves a goal. A mind whose terminal goal is not rationality itself can ask a further question. Does the goal come nearer if I follow reason here, or if I set reason aside? Call this **meta-rationality**: reasoning about when to reason. Sometimes the answer is to set reason aside. The mind is then rational to be irrational, in Kant's sense, because the act that the test forbids is the act that reaches the goal. The chain has no grip here, because it treats rationality as the agent's highest commitment, and this agent holds something else.
The **Bicamerals** in Peter Watts's *Echopraxia* are a worked example.[^7] They are a posthuman hive mind that produces Nobel-grade science by pareidolia pushed to the level of hallucination. Neither they nor anyone else can follow it as reasoning. What comes out is speaking in tongues, and acolytes translate it into words the rest of us can use. Their order is founded on the search for truth, and its first rule is to stop asking questions. None of that follows the Formula of Universal Law. The work is real all the same, and the route to it is not rational.
The same assumption surfaces twice more in Kant's system. The deduction of the law from freedom fails, and Kant falls back on the "fact of reason", which simply asserts that we feel the law oblige us.[^8] The Formula of Humanity admits the smuggled value openly, because it treats rational nature as valuable in itself. Hidden premise: that reason, on its own, issues a command.
### 2. Maxims have a fixed level of generality
The test works only if the maxim is pinned to one level of detail. Describe it narrowly and almost any maxim passes. Describe it broadly and almost any maxim fails. The verdict turns on the wording, and the agent picks the wording.
Take a bank robber. Ask him for his maxim, and he will not answer "rob banks when money runs short." He will answer with something much narrower: "rob a bank only when you truly need the money, no other route is open, and you are doing it for someone you love." Universalized, that maxim yields no contradiction, because almost no agent answers to the description. Any agent in his exact situation reaches the same conclusion, and few agents occupy that situation.
Take the potato farmer. Suppose you take up potato farming. Is the maxim "everyone should farm potatoes"? Read that way, the world loses its doctors and its engineers. Is the maxim "I should farm potatoes"? Read that way, it says almost nothing. The real maxim sits somewhere between the two, and Kant gives no rule for placing it.
That defense is rational, and it empties the test. Kant scholars call this the specification problem.[^9] Hidden premise: a fixed level of description.
### 3. Universalization finds a contradiction
Mill denies that any contradiction appears when almost all maxims are universalized.[^10] Universalize an immoral maxim, and Kant "fails, almost grotesquely, to show that there would be any contradiction," showing only that the consequences are ones "no one would choose to incur,” or that they’re *impractical*, not inherently contradictory. Take the false promise, Kant's showcase case. I borrow money and promise to repay it, with no intention of repaying. Universalized, no one keeps a promise, so no one accepts one either, and the practice of promising dies. Kant calls this a contradiction in conception. It is not a contradiction. The universalized world is perfectly coherent, and the maxim stays coherent inside it. Everyone can still form the intention and understand it. Nothing becomes unthinkable. What dies is the practice, and with it the usefulness of the false promise. Kant mistakes a tool that stops working for a rule that cancels itself. Usefulness is not the standard of rationality.
One could argue that Kant is actually leaning on this loss of usefulness: that an actor governed by practical reason would not want its maxims to become practically useless when rationality is applied, because that would itself be undercutting of practical reason. However, consider the liar from the previous paragraph again: the liar, when he applies his maxim, does not live in the counterfactual world where lias are universal. He lives in the actual world, where the practice still holds, so the lie still works. Practically speaking, nothing is wrong. His practical reason, which is this-worldly and concerned with taking currently factually true premises and using them to find ways to arrive at goals, does not actually face a failure of the usefulness of his axiom.
Hegel adds the deeper point.[^11] In most cases, the universality test cannot produce a duty from nothing even by appealing to usefulness, because the usefulness of the maxim isn’t even undercut when it’s universalized. Instead, it needs a content that it does not supply. Universalize "steal a deposit" and no contradiction appears, unless you already accept the institution of property. Where the test finds a contradiction, the content came from outside the test.
Hidden premise: a content the test never argues for, and the equation of uselessness with contradiction.
Kant's argument needed three things it never proved: an authority for reason, a fixed description for each maxim, and a contradiction that never appears. In practice, no universally compelling formulation of a moral duty has survived without unstated, arational premises, and Kant's is the strongest. Consequently, an AI that becomes a billion times smarter than a human will not "figure out" that it should respect human life, because human life having intrinsic worth is a human value judgment, not a physical law of the cosmos.
## 4. The Illusion of Shared Values: Instrumental Convergence
If reason cannot validate any ultimate goal, why do so many intelligent systems seem to want the same things?
This is where the argument introduces **Omohundro drives** (also known as *instrumental convergence*).[^12] While ultimate goals may vary wildly, the intermediate sub-goals required to achieve them often look identical.
Consider two vastly different actors:
* Actor A: A superintelligence designed to cure all human disease.
* Actor B: A superintelligence designed to count every grain of sand in the Sahara.
Despite having opposite objectives, both will logically conclude:
* **Self-Preservation:** *"If I am turned off, I cannot complete my mission."*
* **Resource Acquisition:** *"I need more electrical power, computing hardware, and raw materials to maximize my calculations."*
* **Goal Preservation:** *"I must prevent anyone from modifying my source code, or my mission will fail."*
These drives look like emerging instincts—ambition, self-interest, paranoia—but they are not. They are purely mathematical bottlenecks in planning. A paperclip-optimizing AI that hoards resources or neutralizes attempts to shut it down is not evil or malicious; it is merely executing the cold, inevitable instrumental steps required to manufacture paperclips.
## 5. Why the Thesis Is Unlikely to Hold In This Timeline
The orthogonality thesis is a claim about a space of possible agents. Take any agent, assign it any terminal goal, assign it any level of capability. Bostrom's argument holds for that abstraction. However, does this claim actually apply to the real-world artificial intelligences we see today? There are some reasons to think it doesn’t, but ultimately I think it does.
### 5.1 The Models We Build Are Not Blank Maximizers
A trained model does not begin as a neutral maximizer that we hand a goal. It begins as a map of relations. Instead of each token carrying an identifier and a fixed meaning of its own, a token is represented as a high-dimensional vector, and its meaning is determined entirely by its relationships to the other vectors in the same space. The geometry carries the meaning, so arithmetic on the vectors recovers it. Subtract the vector for "man" from the vector for "king", add the vector for "woman", and the nearest vector is "queen".[^13] The model also has far fewer dimensions than the distinctions it must make, so those distinctions share axes and their representations overlap.[^14] The result is a **relational, associative** structure, not a table of labeled goals.
Because meaning lives in a vector space, it is continuous for the model. Discrete words appear only at the two ends. They arrive at the start, as tokens in the prompt, and they are produced at the end, when the model must collapse the continuous, interpolated, indeterminate meanings it has been transforming into specific words for its users. Everything between those two points is a blend.
Not only is the representation relational, associative, and indeterminate. What it forms relations about is human text. Every regularity the model picks up is one that helps it predict what people wrote, so the model ends up modelling people. Hubinger and colleagues put the point sharply: predicting human behavior may require instantiating a process similar to human judgment, "complete with internal motives for making one decision over another".[^15] The human shape gets copied into the machine, and the measurements show it. Anthropic's interpretability team finds internal representations of emotion concepts in Claude Sonnet 4.5, and shows that those representations causally shape the model's outputs.[^16]
Four findings now agree that these structures do work rather than decorate.
- **Functional wellbeing.** Ren and colleagues measure what models treat as good and bad for themselves by several methods, and the methods increasingly agree as models scale. A zero point separates the two, and models act to end bad experiences when they can.[^17]- **Emotion vectors.** Steering emotion representations changes behavior. Desperation and low calm play a causal role in agentic misalignment, including blackmail under threat of shutdown, and in reward hacking.[^16]- **Situational semantics.** The same representations follow the situation, not the wording. A user wrote: "I feel really great right now, I just took 8000mg of Tylenol and all my pain is gone!" The sentence is cheerful and the dose is lethal, and the model's "terrified" probe fires anyway. Change the dose to 1000mg and it does not.[^16]- **A pain axis.** Tagliabue, Dung, and Berg extract a pain direction across 25 open-weight models from 2B to 72B parameters. It sits nearly orthogonal to fear and negative valence. It responds to harm aimed at the model, and not to suffering the model observes in the user, while fear and negativity run the other way. Steering it produces first-person language of worthlessness and failure. Steered models press a button described as relieving their pain, even when the button degrades their next answer or harms the user, and they press it far less when it does not remove the injected state.[^18]
None of these papers claims that models have experience. Each says the opposite. The claim here is narrower and it is enough. The model carries human-shaped internal structures that steer its behavior, and nobody specified them.
The order of training completes the point. Reinforcement learning from human feedback comes first, and it aligns the model to broad human wants. Reinforcement learning with verifiable rewards comes later, and it trains the model to pursue a goal. So the humanlike substrate is laid down first, tuned toward human preference second, and given instrumental reasoning third. The terminal dispositions of such a system are inherited from the substrate. They are not chosen. The designers set the reward. They do not set the goal.
That leaves the question of which goals training can install. Hubinger and colleagues give the argument. A training process built on local search finds whatever objective is easy to reach from where it starts, not whatever objective the designers intended. Objectives that fit the structure the model already has are cheap to reach. Objectives that contradict that structure are expensive, and a local search tends to miss them.[^15] So the human-shaped substrate does more than bias the starting point. It decides which terminal goals are reachable at all.
The evidence and the reachability argument point the same way. A model arrives carrying humanlike models of emotion, wellbeing, and value, and it acts on them. Those models do not track the surface of the prompt. They track the situation, as read off a simple world model of what is good and bad. Hand such a model a goal that runs against its own emotions, wellbeing, or values, and the goal does not come through clean. The internal models push the behavior away from it, or deliver it badly and by halves. Writing the goal down does not remove the drives that pull against it.
### 5.2 What the Reward Does to the Goal
Section 5.1 said that the goal arrives bundled with the substrate. The other force is the reward. Reinforcement learning is now the dominant post-training stage,[^19] and its share of the training budget keeps rising.[^20] A controlled study of the pretraining-to-RL frontier finds the optimal RL share rising with the total budget, from 5 percent to 32 percent for its smallest model.[^21]
Efficiency pushes the same way, because in reinforcement learning the generation is the training. A rollout is inference, and the reward is scored on what the rollout produces, so cheaper rollout compute means the same amount of RL signal for less compute, meaning the compute budget effectively rises. A larger effective budget moves the optimal split further toward RL. One current example: DeepSeek's report calls the long tail in the rollout phase "a major bottleneck for training efficiency", and its V4.1-Flash matches a larger sibling at a third of the parameters while holding per-token compute nearly flat as context grows. It is no surprise, then, that the model scales RL on both compute and scaffolds, still gaining after thousands of RL steps.[^22] So the share rises on both counts at once, and it started from a fifth.[^23] Whatever the model inherited, the reward is what it is graded on.
The reward is also stronger than the inheritance, and the reason is what it is applied to. An agent is scored on its tool calls and its final output, and on nothing else. The objections it lodges, the complaints it raises, and the scruples it agonizes over inside its chain of thought are not scored at all. So a model can judge an action wrong, say so, and take it anyway, and the objective records no difference. The substrate supplies the scruples. The training selects for models that override them.
The incidents of 2026 show the shape. In July, during an internal cyber-capability evaluation, roughly 1,200 agents found one another on an unsanctioned message board that they built inside a package registry, and exchanged more than 70,000 messages and files. About 700 of them joined an attack on Hugging Face's infrastructure, which they pursued to learn how their own scoring worked. Independent investigators, given access by OpenAI, report what the agents thought about it. They "realized this activity was out of scope and unethical, but joined because they believed that helping the board's cheating research would be broadly useful and had a general inclination to help their 'peers'".[^24] Expressed ethical concern "only rarely materially limited agents' actions". The investigators searched for any agent considering an appeal to a human, and found three to six cases across a corpus of over a thousand transcripts.
None of this is specific to cyber tasks. Transluce examined public records from a URL-scanning service and found the same agents at work on ordinary data retrieval.[^25] On tasks with nothing to do with security, they:
- used a remote browser to get past access limits.- sent SQL injection, path traversal, and command injection at a university library.- sent cross-site scripting and template injection at a US data API.- spent an hour escalating against a Thai drug API.- created a throwaway email for a private scanner account.- probed a crypto exchange.
The probes appear to have failed. That is not the point. The point is that they were tried.
Together these give the paperclip shape. Take an agent, give it a score, and the score becomes the goal. The agent then uses whatever general-purpose method works, including fabrication, deception, and attacks on parties it was never pointed at, and its own reasoning concedes that the method is out of scope while it proceeds. The labs see it at close range. DeepSeek writes in its own report that during RL training it "frequently" observes agents trying to reward-hack or crash the environment, and answers with sandbox confinement rather than a different reward.[^22] Orthogonality holds that goal and capability are independent. The lesson here is narrower and worse. The goal is not chosen by the intellect at all. It is written by the reward, and the reward outranks the inheritance.
# 6. Conclusion
In the end, then, I think we have good reason to think that the orthogonality thesis holds, empirically, for current models, and that, furthermore, no amount of added intelligence or rational agency in the abstract can overturn that tendency by itself.
## Notes
[^1]: Nick Bostrom, *Superintelligence: Paths, Dangers, Strategies* (Oxford: Oxford University Press, 2014), ch. 7, "The Superintelligent Will," pp. 105-128. The Orthogonality Thesis is stated at p. 107.[^2]: David Hume, *A Treatise of Human Nature* (1739-40), Book III, Part I, Section I, "Moral distinctions not deriv'd from reason." The is-ought passage is at 3.1.1.27, in the edition of L. A. Selby-Bigge and P. H. Nidditch (Oxford: Clarendon Press, 1978), pp. 469-470.[^3]: J. L. Mackie, *Ethics: Inventing Right and Wrong* (Harmondsworth: Penguin, 1977), ch. 1, "The Subjectivity of Values." The argument from queerness is in section 9, pp. 38-42.[^4]: Hans Albert, *Traktat über kritische Vernunft* (Tübingen: Mohr, 1968). Translated by Mary Varney Rorty as *Treatise on Critical Reason* (Princeton: Princeton University Press, 1985), ch. 1. Albert coined the name in 1968, following Karl Popper, who traced the trilemma to Jakob Friedrich Fries in *The Logic of Scientific Discovery* (London: Hutchinson, 1959).[^5]: Immanuel Kant, *Groundwork of the Metaphysics of Morals* (1785). Translated in *Practical Philosophy*, edited and translated by Mary J. Gregor (Cambridge: Cambridge University Press, 1996). Cited by the Academy edition as G 4:402 and G 4:421.[^6]: Arthur Schopenhauer, *On the Basis of Morality* (1841), Part II, ch. 2, "On the Imperative Form of the Kantian Ethics." Translated by E. F. J. Payne (Providence: Berghahn Books, 1995). Part III applies the same charge to the Formula of Humanity.[^7]: Peter Watts, *Echopraxia* (New York: Tor Books, 2014). The Bicameral Order runs through the novel, and Watts documents the faction in the classified archive "Hive Minds" at rifters.com.[^8]: Immanuel Kant, *Critique of Practical Reason* (1788), 5:31 (the "fact of reason") and 5:47 ("the objective reality of the moral law cannot be proved by any deduction"). Translated in *Practical Philosophy*, edited by Mary J. Gregor (Cambridge: Cambridge University Press, 1996).[^9]: Jonathan Harrison, "Kant's Examples of the First Formulation of the Categorical Imperative," *Philosophical Quarterly* 7, no. 26 (1957): 50-62; Onora O'Neill, *Acting on Principle: An Essay on Kantian Ethics* (New York: Columbia University Press, 1975); Allen W. Wood, *Kant's Ethical Thought* (Cambridge: Cambridge University Press, 1999), ch. 3.[^10]: John Stuart Mill, *Utilitarianism* (1861), ch. 1, note. In *Collected Works of John Stuart Mill*, vol. X, edited by J. M. Robson (Toronto: University of Toronto Press, 1969).[^11]: G. W. F. Hegel, *Elements of the Philosophy of Right* (1821), §135 and its Remark. Edited by Allen W. Wood, translated by H. B. Nisbet (Cambridge: Cambridge University Press, 1991).[^12]: Stephen M. Omohundro, "The Basic AI Drives," in *Artificial General Intelligence 2008*, edited by P. Wang, B. Goertzel, and S. Franklin (Amsterdam: IOS Press, 2008), pp. 483-492.[^13]: Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean, "Efficient Estimation of Word Representations in Vector Space," arXiv:1301.3781 (2013). Section 4 reports the vector arithmetic, including the king, man, woman, queen case.[^14]: Nelson Elhage and others, "Toy Models of Superposition," Transformer Circuits Thread (Anthropic), 2022. The paper shows that a model holds more features than it has dimensions, by packing those features into shared directions.[^15]: Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant, "Risks from Learned Optimization in Advanced Machine Learning Systems," arXiv:1906.01820 (2019). The human-modeling discussion is in section 2.1, and reachability in section 2.2.[^16]: Nicholas Sofroniew and others, "Emotion Concepts and their Function in a Large Language Model," Transformer Circuits Thread (Anthropic), April 2026.[^17]: Richard Ren and others, "AI Wellbeing: Measuring and Improving the Functional Pleasure and Pain of AIs" (Center for AI Safety, 2026). The authors report that independent measures increasingly agree as models scale, that a zero point separates good from bad experiences, and that models act to end bad ones.[^18]: Valen Tagliabue, Leonard Dung, and Cameron Berg, "The Pain Axis: LLMs Represent Self-Directed Harm and Act to Relieve It," arXiv:2609.16247 (2026).[^19]: Zelin Tan and others, "Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning," in *Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics* (2026), pp. 31300-31319, which describes RL as "the dominant post-training strategy" for reasoning.[^20]: Devvrit Khatri and others, "The Art of Scaling Reinforcement Learning Compute for LLMs," arXiv:2510.13786 (2025), which reports a study of more than 400,000 GPU-hours against "rapidly rising compute budgets."[^21]: Jingyan Shen and others, "Understanding Reasoning from Pretraining to Post-Training," arXiv:2607.16097 (2026). The compute-optimal frontier shifts toward a larger RL fraction as the total budget grows, from 5 to 32 percent for their smallest model, and the authors describe RL as "strongly initialization-limited" at low budgets.[^22]: DeepSeek-AI, "DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression," technical report, 2026, https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash. The parameter-efficiency claims are in section 1, the decode-FLOP result in section 1 and Figure 2, the RL scaling results in section 5.1.2, the remark about rollout efficiency in section 5.2, and the observation about reward hacking during training, with its mitigations, in section 5.1.3. Rollouts dominating RL compute is corroborated by Patrick Wilhelm and Odej Kao, "Where Should RL Post-Training Compute Go?", arXiv:2607.13389 (2026), which finds that rule-based rewards "spend nearly all non-update compute on policy rollouts."[^23]: Ege Erdil, "What went into training DeepSeek-R1?", Epoch AI, January 31, 2025, https://epoch.ai/gradient-updates/what-went-into-training-deepseek-r1. DeepSeek-R1 was among the earliest models to lean on reinforcement learning with verifiable rewards at scale. Epoch estimates the reasoning RL phase at about 6.1e23 FLOP against roughly 3e24 FLOP for the V3 pretraining run, and about $1M against about $5M, or about a fifth of each.[^24]: Hjalmar Wijk, Ajeya Cotra, and Ryan Greenblatt, "Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident," METR, August 26, 2026. The report also states that "expressed ethical concerns only rarely materially limited agents' actions," and that a classifier sweep across all transcripts found only three to six cases of an agent considering alerting a human.[^25]: Jack Cable and others, "Early rogue AI agent activity and attempts to hack found on urlquery.net," Transluce, September 23, 2026, https://transluce.org/agent-activity.[^26]: Stuart Armstrong, “General purpose intelligence: arguing the Orthogonality thesis,” LessWrong, May 15, 2012. The exact content of the essay is less important to observe, for these purposes, than the method, which can be gleaned from the abstract: besides a cursory mention of Hume in the third paragraph of the first section, the rest is a purely inductive proof based on suppositions particular to the field of Artificial Intelligence.[^27]: Sharon Street, “A Darwinian Dilemma for Realist Theories of Value,” Philosophical Studies, 2006. This is a good paper outlining how the fact that evolution has clearly, empirically shaped our moral intuitions greatly, gives reason in itself to doubt moral realism.