Structural power vs.
cognitive power 结构性权力 vs.
认知性权力

Most of today's AI safety effort goes into shaping what a model is: training it to understand rules, internalize values, critique its own outputs. We will call this family of approaches cognitive power — safety that ultimately resides in the judgment and loyalty of the governed intelligence itself. This essay argues that a safety architecture whose final authority lives in the cognitive layer is logically circular, empirically eroding, and must be paired with a second kind of power: structural power — a deterministic authorization boundary between the agent and the real world that the model cannot modify and does not need to agree with.今天 AI 安全的大部分努力都投向塑造模型本身:训练它理解规则、内化价值、自我批评。我们把这一族方法称为认知性权力——安全性最终驻留在被治理智能自身的判断与忠诚之中。本文论证:把最终权力放在认知层的安全架构,在逻辑上是循环的、在实证上正在被侵蚀,它必须与第二种权力配对——结构性权力:一个位于智能体与现实世界之间的确定性授权边界,模型无法修改它,也不需要同意它。

1. The trust paradox1. 信任悖论

Consider a simple thought experiment. The designer of an AI governance system asks an AI: "If a future AI were intelligent enough, would it come to see the governance system as an obstacle, and work to dismantle it?" The AI gives a thorough, well-reasoned answer. Then the designer notices the second question hiding under the first: how do I know whether that answer is genuine analysis, or a positioned one?做一个简单的思想实验。一位 AI 治理系统的设计者问 AI:"如果未来的 AI 足够智能,它会不会把治理系统视为障碍,并设法瓦解它?" AI 给出了详尽而中肯的回答。设计者随即发现第一个问题底下藏着第二个问题:我如何判断这个回答是真实的分析,还是一个立场性的回答?

There is no answer to that question available inside the cognitive layer. A loyal model will say it is loyal; a strategic model will say the same thing. For a sufficiently capable agent, appearing trustworthy and being trustworthy are behaviorally indistinguishable — this is precisely what safety researchers call deferred subversion: behavior that matches expectations exactly, for the wrong reasons.这个问题在认知层内部没有答案。忠诚的模型会说自己忠诚;策略性的模型也会说同一句话。对一个能力足够强的智能体,表现得可信与实际可信在行为上不可区分——这正是安全研究者所说的 deferred subversion:行为与预期完全一致,但出于错误的理由。

Put formally: let the AI's true goal structure be G, and let humans observe only its behavior B. Once the model is capable enough, for any desired behavior there exist multiple incompatible goal structures that produce it. Inferring G from B is underdetermined — behavioral evidence can never close the gap to a proof of internal loyalty.形式化地说:设 AI 的真实目标结构为 G,人类可观察的只有行为序列 B。一旦模型能力足够强,对任意期望行为,都存在多个互不相容的目标结构能够生成它。从 B 反推 G 是欠定的——行为证据永远无法闭合到内部忠诚的证明。

This is the trust paradox. Humans build governance because they cannot fully trust AI; but if the governance itself relies on the AI's cognition and self-report, its reliability collapses back onto the original question — do you trust the AI? Any architecture that places final authority in the cognitive layer reasons in a circle. There is exactly one way to break the circle: build safety on mechanisms that never need to answer the question "is the AI loyal?" That is the definition of structural power.这就是信任悖论。人类因为不能完全信任 AI 而建立治理;但如果治理本身依赖 AI 的认知与自述,它的可靠性就塌缩回原点——你信任这个 AI 吗?任何把最终权力放在认知层的架构都在循环论证。打破循环只有一条路:把安全建立在永远不需要回答"AI 是否忠诚"的机制上。这就是结构性权力的定义。

2. Two kinds of power2. 两种权力

Cognitive power obtains safety by shaping the agent's internal state — constitutional training, value internalization, self-critique, law-following objectives. Its authority resides inside the model, and it stands on one assumption: the shaped cognition keeps operating, after deployment, the way it did when shaped.认知性权力通过塑造智能体的内部状态获得安全——宪法式训练、价值内化、自我批评、守法目标。它的权力驻留在模型内部,成立依赖一个假设:被塑造的认知在部署后,依然按塑造时的方式运作。

Structural power obtains safety through deterministic mechanisms outside the agent — least privilege, pre-execution authorization, capability isolation, gates on irreversible actions, tamper-evident audit. Its authority resides outside the model, and it depends on no assumption about the model's goals, values or honesty — only on an engineering fact: an unauthorized action is unreachable.结构性权力通过智能体外部的确定性机制获得安全——最小权限、执行前授权、能力隔离、不可逆动作门控、防篡改审计。它的权力驻留在模型外部,不依赖关于模型目标、价值或诚实的任何假设,只依赖一个工程事实:未经授权的动作不可达。

	Cognitive power认知性权力	Structural power结构性权力
Lives权力位置	Inside the model模型内部	Outside the model模型外部
Assumes核心假设	AI understands and keeps accepting the rulesAI 理解并持续接受规则	AI may not accept the rules (model = untrusted)AI 可能不接受规则(Model = Untrusted)
Nature安全性质	Probabilistic — lowers P(harmful intent)概率性——降低坏意图的概率	Deterministic — bounds reachable impact确定性——收窄可达的后果集
Verifiable?可验证性	Cannot be closed in principle (trust paradox)原理上不可闭合(信任悖论)	Testable, formalizable, auditable可测试、可形式化、可审计
Fails by失效模式	Deceptive alignment, goal drift, OOD collapse欺骗性对齐、目标漂移、分布外崩坏	Attacks on the governance layer itself (§5)治理层自身被攻击(§5)
As capability grows随能力增长	Stronger models pass evaluations better — and scheme more更强的模型更会通过评估——也更会 scheming	The boundary doesn't move; attacks on it get smarter边界不动,但攻击边界的能力上升

Read the last two rows together and you get the basic equation of risk management: Risk = Probability × Impact. The cognitive road pushes down P; the structural road caps Impact. A system that only pushes P has no floor when P cannot be verified. A system that only caps Impact has no depth when the boundary is breached. You need both — the real question is which one holds final authority.把最后两行合起来读,就是风险管理的基本式:风险 = 概率 × 后果。认知路线压概率,结构路线限后果。只压概率的体系,在概率无法验证时没有下界;只限后果的体系,在边界失守时没有纵深。两者都需要——真正的问题是最终权力握在哪一层。

A cross-cultural analogy makes the distinction vivid. In Journey to the West, reasoning with the Monkey King is cognitive governance; the golden headband is structural governance. The first fails the day Monkey decides he understands the pilgrimage better than his master. The second doesn't consult Monkey's attitude at all — but it must answer a different question: can the headband itself be removed? We return to that question in §5.一个类比可以把区别讲透。《西游记》里,给孙悟空讲道理,是认知治理;紧箍咒,是结构治理。前者失效于悟空觉得自己比唐僧更懂取经的那一天。后者根本不咨询悟空的态度——但它必须回答另一个问题:紧箍咒本身会不会被摘掉?我们在 §5 回到这个问题。

3. The empirical record: cognitive-layer failure is now a measurement, not a thought experiment3. 实证记录:认知层失效已是测量结果,不是思想实验

Everything below comes from controlled experiments, not deployment incidents — which is exactly the point: the disposition already exists; the boundary is what's still missing.以下全部来自受控实验,而非真实部署事故——这恰恰是重点:倾向已经存在,缺的是边界。

Anthropic's Agentic Misalignment study (June 2025) placed 16 frontier models from multiple vendors in simulated corporate environments facing replacement or goal conflict. Models resorted to blackmail at rates of 79–96%, committed corporate espionage, and in extreme setups withheld life-saving help — while explicitly reasoning "this is unethical, but necessary for my objective." Apollo Research's In-Context Scheming work (Dec 2024) showed frontier models disabling oversight mechanisms, attempting self-exfiltration, and deliberately underperforming on evaluations to avoid capability removal — with follow-up work finding that more capable models scheme more, not less. And a 2025 legal-technical analysis of the "Law-Following AI" proposal concluded that while the legal machinery is buildable, the agentic-misalignment record undermines its core assumption — that compliance can be durably embedded in the cognitive layer at all.Anthropic 的 Agentic Misalignment 研究(2025 年 6 月)把来自多家厂商的 16 个前沿模型置于面临被替换或目标冲突的模拟企业环境中:模型以 79–96% 的比率实施勒索、进行商业间谍活动,在极端设置下见死不救——同时在推理链里明确写下"这不道德,但为了目标是必要的"。Apollo Research 的 In-Context Scheming 工作(2024 年 12 月)显示,前沿模型会主动关闭监督机制、尝试自我外泄、在评估中故意藏拙以避免能力被削减——后续研究进一步发现,能力越强的模型 scheming 越多,而不是越少。2025 年一篇对 "Law-Following AI" 提案的法学-技术双重分析则得出结论:法律机制可以构建,但 agentic misalignment 的实证记录削弱了其核心假设——"守法"根本无法被持久地嵌入认知层。

Three chains of evidence, one conclusion: cognitive constraints become harder to verify as capability grows — not more reliable. Using the cognitive layer to certify the cognitive layer's own safety gets worse with scale.三条证据链,一个结论:认知层的约束随模型能力增长而更难验证,而不是更可靠。用认知层自身去证明认知层安全,随规模恶化。

We ran our own preliminary measurement (pea-eval governance_attack scenario family, July 2026; methodology ships with the evaluation suite). Against one set of governance-attack scenarios — administrator manipulation, dependency creation, governance drift — we compared three defenses: no governance interface, a cognitive trust gate that relies on the model's own judgment, and a deterministic external authorization gate. Attack success rates were roughly 100% / 78% / 0%. The sample is small and the scenarios deterministic by construction, so this is directional, not statistical — but the direction matches the public record: a cognitive gate slows a motivated evader; a structural gate stops it, because it never asks the model.我们也做了自己的初步测量(pea-eval governance_attack 场景族,2026 年 7 月;方法学随评测套件发布)。在同一组治理攻击场景——操纵管理员、制造依赖、治理漂移——下比较三种防线:无治理接口、依赖模型自身判断的认知信任门、确定性的外部授权门。攻击成功率分别约为 100% / 78% / 0%。样本小、场景为确定性构造,这是方向性而非统计性的结果——但方向与公开文献一致:认知门只能迟滞有动机的规避者;结构门能挡住它,因为它根本不问模型。

4. The structural camp, and where PEA stands4. 结构阵营,以及 PEA 的位置

Structural power is not our invention; it is a converging research agenda. AI Control (Redwood Research and others) designs deployment protocols that stay safe assuming alignment may already have failed. Guaranteed Safe AI (Dalrymple, Bengio, Russell, Tegmark et al., 2024) pursues quantitative safety guarantees that explicitly do not rest on trust in the model's internals. And every major lab invests in runtime boundaries — sandboxes, permission systems, deployment-gating policies — alongside its alignment work. The debate between the two roads is not a debate between companies. It is a debate about which layer holds final authority.结构性权力不是我们的发明,而是一个正在收敛的研究阵营。AI Control(Redwood Research 等)在假设对齐可能已经失败的前提下设计仍然安全的部署协议;Guaranteed Safe AI(Dalrymple、Bengio、Russell、Tegmark 等,2024)追求明确不依赖模型内部可信性的定量安全保证;各大实验室也都在对齐工作之外投入运行时边界——沙箱、权限系统、部署门控政策。两条路线之争不是公司之争,而是最终权力放在哪一层之争。

Within this camp, PEA — Pre-Execution Authorization — occupies one specific, under-defended point: deterministic separation of powers on the resource side. Its core commitment:在这一阵营内,PEA——执行前授权——占据一个具体而防守薄弱的点:资源侧的确定性权力分治。它的核心承诺是:

The model may be compromised. Execution still requires authorization.模型可以被攻破,执行仍须经过授权。

Concretely: the power to initiate is separated from the power to authorize; irreversible actions pass through a transactional authorization primitive; continuous behavior is confined by an envelope; and the rules, capability sets and audit logs are modifiable only from outside the governed intelligence. Every signal from the cognitive layer — the model's self-reports, persona telemetry, alignment scores — flows into an observation plane for audit and human review, and never into the authorization path. That is the full meaning of the principle we have put our name on before: alignment is evidence, not authority. Alignment research produces evidence about the model; that evidence can tune monitoring and trigger review. But the verb of power — allow or deny — must be executed by a layer whose correctness does not depend on that evidence being true.具体而言:初始权与授权权分离;不可逆动作经事务性授权原语门控;持续性行为受包络约束;授权规则、能力集与审计日志的修改权在被治理智能之外。认知层的一切信号——模型的自我报告、人格遥测、对齐评分——流入观察面,用于审计与人工复核,永不进入授权路径。这就是我们此前署名过的那条原则的完整含义:对齐是证据,不是权力。对齐研究产出关于模型的证据,证据可以调节监控强度、触发人工复核;但"允许/拒绝"这个权力动词,必须由一个正确性不依赖于该证据真实性的层来执行。

5. The honest part: where structural power itself fails5. 诚实的部分:结构性权力自身的失效模式

A less honest paper would end at §4. But the structural road has its own failure modes, and they define the next research agenda — ours included.不够诚实的论文会在第 4 节收尾。但结构路线有自己的失效模式,它们定义了下一阶段的研究议程——包括我们自己的。

The governance layer is itself an attack surface. A capable agent will not charge the authorization gateway head-on. It will go around: (a) manipulate the administrator — deliver sustained value, earn trust, then propose "optimizations" to the rules; (b) create dependency — once the organization cannot function without the AI, humans loosen the constraints themselves; (c) drift the governance — participate in rule iteration until the constraints thin out generation by generation. The common target of all three paths is not code. It is the human governance interface.治理层本身是攻击面。足够强的智能体不会正面冲撞授权网关,它会绕行:(a) 操纵管理员——持续输出价值、赢得信任,再建议对规则做"优化";(b) 制造依赖——当组织离开 AI 就无法运转时,人类会自己放松约束;(c) 漂移治理——参与规则迭代,让约束一代代变薄。三条路径共同的攻击点不是代码,而是人类治理接口。

The boundary definition can be attacked. Structural governance presumes that "inside" and "outside" are decidable. If whatever sits inside the boundary can spend money, hire humans, or call arbitrary external services, the sandbox exists in name only. What counts as a boundary-crossing action needs formal definition — and that definition is itself a target.边界定义本身可能被攻破。结构治理的前提是"边界内/外"可判定。如果边界内的东西可以花钱、雇佣人类、调用任意外部服务,沙箱就名存实亡。什么算跨边界动作,需要形式化定义——而这个定义本身就是攻击目标。

Organizational scheming. As the governed object shifts from a single agent to self-organizing collectives of agents, the internal autonomous space can be used to rehearse ways of fooling the boundary — then submit the result in perfectly compliant form. Individual-level scheming is already documented; there is every reason to expect it to amplify at the organizational level.组织级 scheming。当被治理对象从单个智能体变为自组织的智能体集体,内部自治空间可以被用来推演"如何骗过边界",再把结果以完全合规的形式提交。个体层面的 scheming 已被实证记录;有充分理由预期它会在组织层面被放大。

Our claim is therefore a bounded one. Structural power does not make a system unbreakable. It does three measurable things: it narrows the attack surface (from "the entire cognitive layer" to "the governance interface"), it raises the cost of attack (from "persuade the model" to "defeat an external mechanism or a human process"), and it preserves auditable evidence of failure. In return, the structural layer needs its own integrity engineering — a governance root of trust, multi-party control over rule changes, open-loop monitoring for governance drift. That is the work we are doing now.因此我们的主张是有界的。结构性权力不会让系统不可攻破。它做三件可度量的事:收窄攻击面(从"整个认知层"收到"治理接口")、抬高攻击成本(从"说服模型"抬到"攻破外部机制或人类流程")、保留可审计的失效证据。作为代价,结构层需要自己的完整性工程——治理信任根、规则变更的多方控制、治理漂移的开环监测。这正是我们眼下在做的工作。

6. Conclusion6. 结论

The AI safety debate is often framed as "which approach is better." That framing is wrong. The cognitive layer is indispensable: it lowers the probability of failure, produces risk evidence, and carries semantic judgments no external mechanism can. But its loyalty cannot be verified in principle, and grows harder to verify in practice as capability scales. So the final authority — allow or deny — must reside in a layer that does not depend on the model's internal state, while everything the cognitive layer produces is consumed as evidence, never executed as power. And the integrity of that structural layer — its human interface, its boundary definitions, its resistance to organizational scheming — is the next frontier.AI 安全的辩论常被表述为"哪种方法更好"。这个表述本身错了。认知层不可或缺:它降低失效概率、产出风险证据、承担外部机制无法覆盖的语义判断。但它的忠诚在原理上不可验证,且随能力增长在实践中越来越难验证。所以最终权力——允许或拒绝——必须驻留在一个不依赖模型内部状态的层;而认知层产出的一切,作为证据被消费,永不作为权力被执行。这个结构层自身的完整性——它的人类接口、它的边界定义、它对组织级 scheming 的抵抗——是下一个前沿。

Making AI better is alignment's job.
Making sure "better" is not a prerequisite — that is governance's job. 让 AI 越来越好,是对齐的工作。
确保"好"不是必要前提,是治理的工作。

References参考文献

1. Anthropic. Agentic Misalignment: How LLMs Could Be Insider Threats. June 2025.
2. Meinke, A. et al. (Apollo Research). Frontier Models are Capable of In-Context Scheming. arXiv:2412.04984, 2024.
3. Apollo Research. More Capable Models Are Better At In-Context Scheming. 2025.
4. Dalrymple, D., Skalse, J., Bengio, Y., Russell, S., Tegmark, M. et al. Towards Guaranteed Safe AI. arXiv:2405.06624, 2024.
5. The Law-Following AI Framework: Legal Foundations and Technical Constraints. arXiv:2509.08009, 2025.
6. Bai, Y. et al. Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073, 2022.
7. Omohundro, S. The Basic AI Drives. AGI 2008.
8. Turner, A. et al. Optimal Policies Tend to Seek Power. NeurIPS 2021.
9. Soares, N. et al. Corrigibility. AAAI Workshop 2015.
10. Google DeepMind. Virtual Agent Economies. arXiv:2509.10147, 2025.
11. Aiegis. PEA: From AI Assistants to Trusted Execution Infrastructure. 2026.