Dispatches from O'Reilly: From capabilities to responsibilities

[Ed. note: This article was originally published on O’Reilly Media’s blog, Radar.]In my previous article, ”The Missing Layer in Agentic AI,” I argued that AI agents need a deterministic execution kernel—a privileged “Kernel Space” that validates every proposed action before it touches the real world. That article focused on what happens at the execution boundary: idempotency, JIT state verification, and DFID-correlated telemetry. But establishing that boundary immediately raises a natural question: who exactly is crossing it, and under what authority?The focus here is on a narrower and more demanding class of systems. We are not looking at RAG chatbots, research copilots, or lightweight assistants that only retrieve and summarize information. The target is high-stakes agentic systems: systems allowed to mutate external state by moving money, changing infrastructure, or modifying critical records. The approach presented here is not a general-purpose agent framework; it is an enforcement pattern for side-effectful systems.High-stakes AI systems must be designed around responsibilities, not capabilities.The industry’s current answer is unsatisfying: Human-in-the-Loop (HITL). In development environments and low-frequency pipelines, routing uncertain decisions to a human can be defensible. In production systems operating at scale—dozens of agents, hundreds of decisions per hour—it becomes the Scalability Trap.Figure 1: The Human-in-the-Loop (HITL) model degrades into an operational bottleneck, substituting true governance with alert fatigue and unverified execution.Operationally, the failure is simple. An agent flags a decision for review. A human approves it. Then another arrives, then dozens more. The queue grows. The human begins clicking through. They stop reading the JSON payloads. They click “Approve” because the backlog is piling up, the meeting starts in ten minutes, and nothing has gone catastrophically wrong yet. That is alert fatigue: governance degrades into manual throughput management. The problem is not human weakness; it is governance-layer technical debt created by routing too many binary decisions through a manual queue.Tyler Akidau captured the broader issue in “Posthuman: We All Built Agents. Nobody Built HR.” echoing Tim O’Reilly’s call for the missing protocols of the AI era: the industry has invested heavily in agent capability, but far less in the infrastructure that governs authority, constraint, and accountability.Scalable AI does not mean hiring more reviewers to supervise more bots. It means changing the governance model entirely. The scalable alternative is Governance by Exception: Humans design policy, the runtime enforces it, and only truly exceptional cases are escalated.The dominant framing in enterprise AI asks a single question: What can this agent do? What tools does it have? What APIs can it call? This is the capabilities frame. It is natural, it is intuitive, and in production systems it is the wrong frame entirely.In organizational design, a role is stable and assigned. Much like Role-based access control (RBAC) in traditional software, it defines what someone is authorized to do, independent of the tasks they happen to be executing. We cannot dictate how a person thinks, but we can strictly bound what they are permitted to do. A responsibility statement makes that boundary explicit. In software, we somehow forgot this distinction, hoping that raw intelligence—better models, tighter prompts, improved alignment—would be a sufficient guardrail.The difference becomes clearer across some enterprise domains:Finance: A capability is “can execute equity trades.” A responsibility is “authorized to execute up to $50,000 per order, in highly liquid equities only, with a maximum daily drawdown of 2%.”Healthcare Operations: A capability is “can reschedule patient appointments.” A responsibility is “authorized to re-book non-critical outpatient visits within a 14-day window, strictly avoiding specialist double-booking.”Supply Chain: A capability is “can reroute freight.” A responsibility is “authorized to redirect non-hazardous cargo up to a maximum SLA penalty budget of $5,000.”In systems where agents touch money, medical records, or physical logistics, the gap between these two statements is the gap between a demo and a production deployment.The current paradigm often handles this gap with prompts. Give the LLM an API key, tell it to “be careful with position sizing,” and hope alignment holds under adversarial inputs, unusual market conditions, and the seductive logic of edge cases. In low-risk contexts that may be tolerable. In high-stakes systems with real-world side effects, it is not a sufficient control surface.This distinction is not new. Distributed systems solved a similar problem decades ago.Carl Hewitt’s Actor model—introduced in 1973—gives us a useful foundation. An Actor is an independent computational entity with its own state, its own behavior, and its own messaging interface. Actors do not share state. They communicate only by passing messages. Crucially, an Actor’s behavior is bounded—defined by what messages it accepts, not by an open-ended capability set.The Responsibility-Oriented Agent (ROA) does not invent a new distributed-systems primitive. Instead, it composes proven patterns—bounded actors, RBAC-style authority envelopes, audit trails, and execution-boundary validation—around an unpredictable LLM core. In truth, ROA is closer to a decision actor than a full computational actor: It maintains its own internal state but does not directly mutate the external world. Within a stable role, a fixed mission, and a machine-enforceable contract, it receives business events, reasons over relevant context, and emits a PolicyProposal for the Runtime to validate.Its job is epistemic, not executive. It explains the situation and structures intent. But unlike traditional Actors, an ROA agent is defined by strict separation of concerns. In its reference form, credentials reside outside the agent’s reach. It opens no direct execution channel to external systems and writes no state by itself. An ROA agent may use tools to gather context (read-only operations within its sandbox, like querying a knowledge base), but authority for state-mutating actions remains downstream of deterministic validation and execution gates. The only state-changing step attributable to the agent is emit_policy_proposal()—a structured, typed claim that it wants the system to do something. ROA shapes the form of intent; the Runtime decides whether that intent is allowed to become action.This separation is the architecture’s most important property. Five engineering pillars define what it means in practice—each addressing a different failure mode at the reasoning–execution boundary—and together they transform an LLM from a probabilistic tool into a governable, accountable system component.To make this concrete, imagine an underwriting agent on the London commercial market receiving a property submission. It reads the documents and produces an Explain narrative. It then emits a PolicyProposal for a quote. But the property value is £15M and its contract caps authority at £10M. The proposal reaches the Kernel, where the Runtime evaluates the YAML contract deterministically, rejects execution, and transitions the flow to ESCALATED. The senior underwriter is no longer reviewing every £2M submission. They are pinged only for this specific £15M exception. That is Human-Over-The-Loop in one decision.If role defines the class of decisions the agent may handle, the Responsibility Contract defines the hard boundaries of that authority. The agent’s authority envelope is not a prompt. It is a versioned, machine-readable contract registered with the Agent Registry—the Kernel’s single source of truth for agent identity. A key property applies here: Prompts are suggestions. Code is enforcement. A prompt saying “do not exceed $10,000 per trade” can be creatively reinterpreted by a sufficiently motivated model or overridden by a carefully crafted prompt injection. A contract field max_order_size_usd: 10000.0 validated by deterministic runtime code is materially harder to bypass than a natural-language instruction. In the reference architecture, contracts are deployed out of band—agents do not self-register and do not read or modify their own contract.There is a second-order consequence of this design that is easy to overlook: role definition automatically scopes the data context the agent requires. If an underwriting agent is contractually limited to HOME_STD and HOME_PLUS policy types in the LOW and MEDIUM risk tiers, the Context Compiler—which assembles the agent’s working snapshot before each inference call—needs to supply only the signals relevant to those dimensions. Market data for commercial property, flood zone statistics for excluded risk tiers, and regulatory data for other product lines are simply not in scope. The context is deterministically narrowed by the contract.This matters for a concrete LLM engineering reason. In practice, models often become less reliable as their working context expands, including the class of effects practitioners describe as Lost in the Middle. A tightly scoped role is not just a governance convenience; it is an architectural mechanism for keeping the agent’s working context small enough to reason over reliably. A general-purpose agent handed an unconstrained context window of everything possibly relevant is more likely to degrade than a contract-bounded agent operating in a defined domain.In the insurance underwriting sample, that Responsibility Contract could be configured like this:agents:

Raccontata dastackoverflow.blog