How to Build Reliable Agents With Ai2's Shippy Architecture

The Allen Institute for AI (Ai2) published a technical deep-dive detailing the architectural lessons learned from building Shippy, a specialized AI agent for maritime domain awareness. Released on July 15, 2026, the report outlines a specific framework for handling complex, high-stakes environments where incorrect data could misdirect physical resources like patrol vessels. You can apply this architecture to your own deployments by separating agent logic into distinct components, strictly enforcing deterministic execution boundaries, and running all operations inside isolated infrastructure.

Core Architecture Components

The Shippy architecture avoids monolithic prompts by dividing the agent into three highly specialized, version-controlled components: Soul, Skills, and Config. This separation allows you to update an agent’s technical capabilities without altering its core behavioral guardrails.

The Soul component acts as the foundational system prompt. It establishes the agent’s persona, strict behavioral constraints, and decision-making framework. This file dictates how the agent should react to ambiguity, when it should ask for human intervention, and the specific formats it must use when communicating outputs.

Skills are encoded as versioned, plain-text markdown files. Each file defines a specific workflow and provides direct links to the necessary scripts or internal tools required to execute that workflow. By keeping agent skills in markdown format, you can easily track changes in version control systems and allow human analysts to audit the exact procedures the agent is authorized to follow.

The Config component dictates the runtime settings. It specifies the underlying harness and the model deployed for the current task. Shippy relies on Claude Opus 4.6 and runs entirely on the open-source OpenClaw framework. Decoupling the model configuration from the logic allows you to test different models without rewriting the agent’s core instructions.

Enforcing Determinism With a CLI

Allowing an LLM to generate raw API calls directly introduces significant reliability issues. Models frequently hallucinate parameters, misinterpret complex JSON schemas, or fail to handle API version changes gracefully. To solve this, the Shippy architecture abandons direct API access in favor of a purpose-built Command Line Interface.

Instead of crafting JSON payloads, the agent interacts with backend services through this deterministic CLI. The CLI features strictly typed commands and structured flags. If the agent attempts to use an invalid flag or an incorrect data type, the CLI immediately throws a structured error back to the agent for correction.

This approach provides multiple benefits for operational reliability. It forces the LLM to output simple, verifiable command strings. It also allows your engineering team to thoroughly test the execution layer independently of the LLM. You can write standard unit tests for the CLI commands to ensure they interact with your APIs correctly, removing the stochastic nature of the LLM from the execution phase.

Isolated Infrastructure via Mothership

Security and data privacy are critical when deploying agents that handle sensitive information. For Shippy, which serves hundreds of agencies across 70 countries, Ai2 built an infrastructure layer named Mothership. You should replicate this isolation model when building multi-tenant agent systems.

Mothership provisions an ephemeral, fully isolated Kubernetes sandbox for every individual user session. When an analyst initiates a task, the agent spins up within this dedicated environment. All data processing, tool execution, and temporary file storage occur strictly within the sandbox boundaries.

Once the session ends, the Kubernetes pod is destroyed. This ephemeral model ensures that no sensitive data, context windows, or execution logs bleed between different organizational environments. It also prevents a compromised agent from accessing a shared file system or network layer.

Whole-Agent Evaluation

Static benchmarks fail to capture how an agent performs over time in dynamic environments. The Shippy team replaced standard static testing with a whole-agent evaluation pipeline that tests the system against live, shifting data feeds.

Instead of automated static grading, domain experts write weighted rubrics for real-world scenarios. Every new version of the agent must pass these human-aligned benchmarks before deployment. The testing pipeline feeds the agent live satellite imagery and GPS signals, grading not just the final output but the specific steps and tools the agent used to reach its conclusion.

When evaluating AI agents for production, you should focus your evaluation pipeline on the agent’s ability to navigate ambiguous live data rather than its ability to answer static questions.

Verifiability and Source Linking

Agents deployed in high-stakes environments must allow human operators to verify their work. Shippy mandates that every assertion includes deep links to the original data sources, exact data cutoffs, and map coordinates.

Configure your agents to append a structured citation block to every response. If an agent executes a script to filter data, the response must include a link to the exact log of that script execution. By designing transparency directly into the output format, you give human operators the confidence to act on the agent’s analysis.