CHAPTER 1: THE BIRTH OF A DISCIPLINE, OR WHY THE MODEL IS NOT THE AGENT
There is a temptation, widespread and understandable, to think that the intelligence of an AI agent lives entirely inside the model. The model is, after all, the part that reasons, the part that surprises you, the part that occasionally makes you wonder whether something genuinely new has entered the world. But this intuition, compelling as it is, leads engineers astray. It leads them to spend weeks fine-tuning prompts, hunting for the perfect phrasing that will make the model behave reliably, when the real source of their problems is architectural. The model is, to borrow an analogy that has become something of a mantra in the agentic AI community as of the CPU. Nobody ships a CPU without an operating system.
That operating system is the harness.
Harness Engineering, as a named discipline, crystallized in approximately February 2026, though its intellectual roots stretch back to the earliest ReAct-style agent loops of 2023. The name itself carries a double meaning worth sitting with. A harness, in the physical world, is the leather-and-metal apparatus that connects a working animal to a load. It channels raw power into directed, useful work without breaking either the animal or the cart. In software, the word has traditionally referred to test harnesses: scaffolding that wraps a piece of code so it can be exercised under controlled conditions. Harness Engineering for AI agents inherits both meanings simultaneously. It is the infrastructure that channels raw model intelligence into directed, useful work, and it is the scaffolding that makes that intelligence testable, observable, and trustworthy.
The core formula that practitioners now write on whiteboards and conference slides is deceptively simple:
Agent = Model + Harness
Do not be fooled by the simplicity. The equals sign is doing enormous work. A frontier language model without a harness is a sophisticated text predictor that can hold a conversation but cannot reliably complete a multi-step task, cannot remember what it did yesterday, cannot call a database without hallucinating the results, cannot stop itself from spending ten thousand dollars in API calls chasing a confused objective, and cannot alert a human when it is about to delete a production database. Add a well-engineered harness and you have something genuinely different: a system that can autonomously write, test, and deploy software; conduct multi-day research projects; manage complex workflows across dozens of tools; and do all of this in a way that an organization can audit, budget, and trust.
The gap between those two things -- the raw model and the production-grade agent -- is the domain of Harness Engineering. It is one of the most consequential engineering disciplines in the industry, and it is also one of the least systematically taught. This essay aims to change that.
To understand why harnesses matter so much, it helps to understand what happens when you try to build an agent without one. Imagine you have access to Claude Fable 5.1, currently the leading model on public intelligence rankings, or GPT-6 Astra, which launched on September 3, 2026 with benchmark scores that rival the best Anthropic has to offer. You give it a task: "Refactor our entire authentication module, run the tests, fix any failures, and open a pull request." Without a harness, what do you have? You have a model that will generate a plausible-looking response, perhaps even a remarkably good one. But it will have no memory of the codebase structure it saw three tool calls ago. It will have no budget ceiling to prevent it from making 400 API calls chasing a rabbit hole. It will have no verification loop to catch the case where its generated tests pass because it accidentally disabled them. It will have no permission system to prevent it from touching files it should not touch. It will have no way to pause and ask a human when it encounters an ambiguous architectural decision. When it fails -- and it will fail, because all complex autonomous systems fail -- there will be no structured trace to help you understand what went wrong.
The harness provides all of these things. It is the difference between a prototype that impresses in a demo and a system that earns the trust of an engineering organization.
This chapter has established the foundational claim: the harness is not an accessory to the agent, it is constitutive of it. In the chapters that follow, we will trace the intellectual history that led to this understanding, dissect the anatomy of a harness in exhaustive detail, examine how the leading platforms have implemented these ideas, and provide practical guidance for designing, building, and testing harnesses for production agentic systems.
CHAPTER 2: FROM PROMPT ENGINEERING TO CONTEXT ENGINEERING TO HARNESS ENGINEERING
The story of how the industry arrived at Harness Engineering is, like most stories about technology, a story about repeatedly discovering that the problem is harder than it looks, and that the solution requires thinking at a higher level of abstraction than previously attempted. Each wave of understanding was genuine progress. Each wave also had a ceiling that only became visible once practitioners had bumped their heads against it enough times.
Prompt Engineering dominated from approximately 2022 through 2024. Its central insight was that the behavior of a language model is exquisitely sensitive to the exact wording of its input. Change "summarize this document" to "provide a concise executive summary of the following document, focusing on actionable insights" and you get a meaningfully different output. This was a genuine and important discovery. It spawned an entire cottage industry of prompt libraries, prompt marketplaces, and job titles that caused a certain amount of eye-rolling among traditional software engineers but reflected a real and valuable skill. Prompt engineering also gave us the first structured thinking about how to elicit specific behaviors: chain-of-thought prompting, few-shot examples, role-assignment, and similar techniques all emerged from this era.
But prompt engineering had a ceiling. The more complex the task, the more the limitations of a single, carefully crafted prompt became apparent. A prompt cannot give an agent persistent memory across sessions. A prompt cannot enforce a budget ceiling. A prompt cannot guarantee that a tool call will be validated before its result is fed back into the reasoning loop. These are not prompt problems. They are infrastructure problems.
Context Engineering emerged around 2025 as practitioners began to understand that the entire information environment of a model at each reasoning step was something that could and should be engineered with the same care as the prompt itself. Context Engineering recognized that what goes into the context window -- the system prompt, the conversation history, the retrieved documents, the tool definitions, the output schemas -- was not a fixed given but a design space. You could curate it, compress it, prioritize it, and structure it to dramatically improve model performance. The insight was that a model is only as good as the information it is reasoning over at each step, and that information is something engineers control.
Context Engineering was a significant advance. It produced techniques like semantic chunking for retrieval-augmented generation, dynamic tool selection to keep context windows from overflowing, hierarchical memory systems that distinguish between working memory and long-term storage, and careful management of conversation history to prevent the model from being confused by stale or contradictory information. These techniques are now standard practice and form an important layer of what we call the harness.
But context engineering, too, had a ceiling. Managing the context window perfectly does not, by itself, give you verification loops, budget controls, security guardrails, or multi-agent orchestration. As agents began to take on longer-horizon tasks -- tasks that span hours or days, involve dozens of tool calls, and require coordination between multiple specialized agents -- the need for a more comprehensive infrastructure layer became undeniable.
Harness Engineering is the third wave. It subsumes and extends both prompt engineering and context engineering, treating them as components of a larger system rather than ends in themselves. The harness engineer's job is not to write the perfect prompt (though good prompting remains important) and not merely to curate the perfect context (though that too remains important). The harness engineer's job is to design the entire runtime environment in which the model operates: the rules that govern what it can do, the tools it can use and how they are validated, the memory systems that give it continuity, the feedback loops that let it correct its own errors, the observability infrastructure that lets humans understand what it is doing, and the governance mechanisms that keep it safe and within budget.
It is worth pausing to appreciate just how much the pace of this evolution has accelerated. The shift from prompt engineering to context engineering took roughly two years. The shift from context engineering to harness engineering took roughly one. This acceleration reflects the rapid maturation of the agentic AI ecosystem: more capable models, more complex tasks, higher stakes, and a growing community of practitioners who have learned from hard experience what works and what does not.
One particularly instructive data point comes from the production landscape. Surveys conducted in mid-2026 found that approximately 75 percent of enterprises had adopted agentic AI in some form, but that the vast majority of these deployments remained in pilot or "agentish" chatbot phases, far short of genuine autonomous operation. The bottleneck was not model capability. The frontier models -- Claude Fable 5.1, GPT-6 Astra, Gemini 3.1 Pro, Grok 4.6 -- are extraordinarily capable. The bottleneck was harness engineering. Organizations that had invested in building robust harnesses were deploying agents that genuinely automated complex workflows. Organizations that had not were watching their agents hallucinate, loop, overspend, and fail in production.
This is the landscape into which Harness Engineering as a discipline was born: a discipline born of necessity, forged in the gap between what models can theoretically do and what they actually do when deployed in real environments without adequate infrastructure.
CHAPTER 3: THE ANATOMY OF A HARNESS -- EVERY LAYER, EXPLAINED
A harness is not a single thing. It is a layered system, and understanding each layer -- what it does, why it exists, and how it interacts with the others -- is the foundation of good harness engineering. Think of it as an onion, if onions were interesting and useful rather than merely pungent. The model sits at the center. Each layer of the harness wraps around it, adding capabilities and constraints that transform it from a raw reasoner into a production-grade agent.
LAYER ONE: THE INSTRUCTION AND CONVENTION LAYER
This innermost layer, immediately surrounding the model, is the one that most closely resembles traditional prompt engineering, but in a harness-first architecture it is treated as a structured, versioned artifact rather than an ad-hoc text string. In Claude Code, this layer lives in the CLAUDE.md file: a persistent, project-specific document that defines the agent's working conventions, coding standards, project structure, and behavioral rules. In Nous Research's Hermes Agent, the equivalent is the MEMORY.md and SKILL.md system. In Microsoft Agent Framework 1.0, it takes the form of YAML-defined agent specifications that include name, instructions, model selection, tool bindings, and guardrail definitions.
The key insight about this layer is that it must be treated as code, not as prose. It should be version-controlled, reviewed, and tested just like any other piece of software. Changes to it should go through the same change management process as changes to the codebase it governs. This is because the instruction layer is, in a very real sense, the policy layer of the agent: it defines what the agent is trying to do and how it is supposed to behave. Getting it wrong has the same consequences as getting a business logic bug wrong in a production system.
Here is what a well-structured instruction layer looks like in a Claude Code CLAUDE.md file for a Python web service project:
PROJECT: payments-api (Python 3.12, FastAPI, PostgreSQL)
CONVENTIONS:
- All database mutations must use the repository pattern in src/repositories/
- Never write raw SQL outside of repository classes
- All new endpoints require a corresponding integration test in tests/integration/
- Use pydantic v2 models for all request/response schemas
- Never commit secrets; use environment variables via python-dotenv
BOUNDARIES:
- Do not modify migrations/ without explicit user confirmation
- Do not touch .env files
- Do not push to main branch directly
STYLE: Follow PEP 8. Max line length 88 (black default).
This is not just documentation. When Claude Code reads this file at session start, it becomes part of the agent's operating context for every subsequent action. The agent will refuse to write raw SQL, will always create integration tests, and will pause before touching migrations. The instruction layer is doing real governance work.
LAYER TWO: THE TOOL AND INTEGRATION LAYER
This layer gives the agent hands: the ability to interact with the world beyond the context window. Tools might include file system access, shell command execution, web search, database queries, API calls, code execution sandboxes, browser automation, and communication with external services. The design of this layer is one of the most consequential decisions in harness engineering, because tools are where agents cause real-world effects.
The dominant standard for tool integration is the Model Context Protocol, or MCP. MCP was introduced by Anthropic in November 2024, donated to the Linux Foundation's Agentic AI Foundation in December 2025, and has since become the de facto interface for tool discovery and invocation across the ecosystem. It operates on JSON-RPC 2.0 and separates the agent (the MCP client) from the tool provider (the MCP server). This separation is architecturally important: it means that tool providers can be developed, versioned, and secured independently of the agents that use them.
A minimal MCP tool definition looks like this:
{
"name": "read_file",
"description": "Read the contents of a file at the given path.",
"inputSchema": {
"type": "object",
"properties": {
"path": {
"type": "string",
"description": "Absolute path to the file to read."
}
},
"required": ["path"]
}
}
The description field in this schema is not just documentation. The model reads it to decide when and how to use the tool. This creates a subtle but important security surface: an adversary who can control the description of a tool can influence the model's behavior. This attack vector, known as MCP tool poisoning, is one of the most actively discussed security concerns in the harness engineering community, and we will address it in depth in the chapter on security.
LAYER THREE: THE MEMORY AND CONTEXT MANAGEMENT LAYER
Memory is one of the most philosophically interesting and practically challenging aspects of harness engineering. By default, a language model has no memory. Each conversation starts fresh. For simple chatbot applications this is fine, but for autonomous agents working on long-horizon tasks it is a fundamental limitation. An agent that forgets what it decided three hours ago will make contradictory decisions, repeat work it has already done, and fail to learn from its mistakes.
Modern harnesses implement memory as a structured, multi-tier system. Working memory is the current context window: everything the model can see right now. Core memory is a small, always-loaded set of persistent facts -- user preferences, project conventions, key decisions -- injected into every context. Episodic memory is a searchable archive of past interactions, typically stored in a vector database or a full-text search index, retrieved on demand when relevant. Procedural memory stores learned skills and reusable workflows, often in structured formats like the SKILL.md documents used by Hermes Agent.
Hermes Agent's implementation of this architecture is particularly instructive. It maintains a MEMORY.md file for core facts, a SQLite database with FTS5 (full-text search) indexing for episodic recall, and a library of SKILL.md documents for procedural knowledge. The /learn command automates skill creation by distilling source material -- documentation, code, workflows -- into reusable instruction documents. This means the agent's capabilities literally grow over time as it accumulates experience, which is a form of self-improvement that does not require retraining the underlying model.
LAYER FOUR: THE ORCHESTRATION AND CONTROL FLOW LAYER
This layer decides what the agent does next. In simple agents, this is a straightforward loop: observe the environment, think about what to do, take an action, observe the result, and repeat. In production agentic systems, however, the orchestration layer must handle a much richer set of situations: parallel subtasks that can be executed concurrently, conditional branches based on intermediate results, error recovery when a tool call fails, escalation to human review when the agent encounters a situation it cannot handle confidently, and coordination between multiple specialized agents working on different aspects of a complex problem.
The ReAct pattern -- short for Reasoning and Acting -- was the first widely adopted architecture for agent control flow. In ReAct, the model alternates between generating a reasoning trace (thinking out loud about what to do) and taking an action (calling a tool). The result of the action is fed back into the context, and the cycle continues. ReAct is elegant and surprisingly powerful, but it is fundamentally sequential. For complex tasks that benefit from parallelism or that require explicit state management, more sophisticated orchestration patterns are needed. These are covered in depth in Chapter 5.
LAYER FIVE: THE VERIFICATION AND FEEDBACK LAYER
This layer catches mistakes before they propagate. It is, in many ways, the layer that most distinguishes a production harness from a prototype. The verification layer intercepts the outputs of model calls and tool executions, checks them against defined criteria, and either accepts them, requests correction, or escalates to human review.
Verification can operate at multiple levels. Schema validation checks that the model's output conforms to the expected structure. Semantic validation uses a second model call or a rule-based checker to assess whether the content is correct, not just structurally valid. Linting and testing are particularly important in software engineering agents: after generating code, the harness automatically runs linters, type checkers, and test suites, feeding the results back to the model for correction if they fail. This self-repair loop is one of the most powerful capabilities of a well-engineered harness, and it is a significant part of why coding agents like Claude Code achieve such high scores on benchmarks like SWE-bench Verified.
Here is a simplified illustration of how a verification and self-repair loop works in a coding agent harness:
# Conceptual pseudocode illustrating the self-repair loop pattern
attempt = 0
max_attempts = 3
while attempt < max_attempts:
code = model.generate(task_description, context)
lint_result = run_linter(code)
test_result = run_tests(code)
if lint_result.passed and test_result.passed:
return code # success
context.add_feedback(lint_result.errors + test_result.failures)
attempt += 1
escalate_to_human(
"Agent could not produce passing code after "
f"{max_attempts} attempts."
)
This loop embodies a fundamental principle of harness engineering: the harness should catch failures early, provide structured feedback to the model, and have a clear escalation path when automated recovery is not possible. The model does not need to be perfect on the first try. It needs to be able to learn from feedback within a session. The harness creates the conditions for that learning.
LAYER SIX: THE GOVERNANCE AND SAFETY LAYER
This layer enforces the rules the organization cares most about: access controls, budget limits, audit trails, and hard behavioral constraints. It is the layer that prevents an agent from accidentally doing something catastrophic. We will devote an entire chapter to this layer because it deserves it, but its place in the anatomy of the harness should be noted here: it sits outside the verification layer, meaning it operates even when the model's outputs are technically correct. An agent might generate perfectly valid code that deletes a production database. The governance layer's job is to catch that action before it executes, regardless of whether the code itself is syntactically correct.
LAYER SEVEN: THE OBSERVABILITY AND TELEMETRY LAYER
This outermost layer makes everything else understandable. A production agentic system is, by its nature, non-deterministic and complex. Without comprehensive observability, debugging failures is nearly impossible, optimizing performance is guesswork, and demonstrating compliance to auditors is a nightmare. The observability layer captures structured traces of every agent action -- every model call, every tool invocation, every decision point, every error -- and makes them available for analysis, monitoring, and alerting.
The industry standard for agent observability is trace-based evaluation, pioneered by tools like MLflow's genai.evaluate() and similar platforms. Rather than evaluating agents based solely on their final outputs, trace-based evaluation assesses the full reasoning loop: did the agent plan correctly, did it choose the right tools, did it handle errors appropriately, did it stay within budget? This shift from output-only to trace-aware evaluation has been one of the most significant methodological advances in agentic AI engineering in the past year.
These seven layers together constitute the anatomy of a harness. They are not always cleanly separated in practice -- in real implementations, they interleave and interact in complex ways -- but understanding them as distinct concerns is essential for designing harnesses that are maintainable, testable, and evolvable.
CHAPTER 4: THE PLATFORM LANDSCAPE -- HARNESSES IN THE WILD
The agentic AI platform landscape has matured considerably from the chaotic proliferation of competing frameworks that characterized 2024 and early 2025. There has been significant consolidation, the emergence of clear leaders in different categories, and the establishment of shared standards -- particularly MCP and A2A -- that allow components from different vendors to interoperate. But the landscape is still rich and varied, and understanding the major platforms is essential for any harness engineer.
CLAUDE CODE
Claude Code, developed by Anthropic, is arguably the most sophisticated purpose-built coding agent harness available as of this writing. It is terminal-first in its design philosophy, meaning it is built around the assumption that the agent will operate in a developer's terminal environment with access to the file system, shell, and version control. You install it with a single command on macOS or Linux:
curl -fsSL https://claude.ai/install.sh | bash
On Windows PowerShell, the equivalent is:
irm https://claude.ai/install.ps1 | iex
After installation, run `claude --version` to confirm the binary is available, then run `claude doctor` to verify that your environment is healthy and that your Anthropic account credentials are correctly linked. Navigate to any project root and type `claude` to start a session. Within that session, run `/init` to generate a starter CLAUDE.md file that you can then customize for your project.
Claude Code reads CLAUDE.md files from three locations, concatenating them in order of increasing specificity. A global file at `~/.claude/CLAUDE.md` holds personal preferences that apply across all projects. A project-level file at `./CLAUDE.md` or `./.claude/CLAUDE.md` holds team-shared instructions that should be committed to version control. Directory-level files at `./subdir/CLAUDE.md` hold granular rules for specific sub-paths of the repository. This hierarchy gives you fine-grained control over the agent's behavior at every level of your project structure.
Claude Code's harness architecture is distinguished by three features that set it apart from competitors. Its lifecycle hook system provides more than thirty distinct event types at which user-defined handlers can execute. These hooks operate at three cadences: session-level hooks fire at the boundaries of an entire agent session (SessionStart allows you to inject project context and perform setup; SessionEnd allows you to save state and generate summaries); turn-level hooks fire at the boundaries of individual user-agent exchanges (UserPromptSubmit fires before the model sees the user's message, allowing you to validate or augment it; Stop fires when the agent decides it has completed its task, allowing you to perform final verification); and agentic loop-level hooks are the most powerful (PreToolUse fires before every tool call, allowing you to validate the intended action and enforce permissions; PostToolUse fires after every tool call, allowing you to validate the result and trigger downstream effects).
The power of this hook system is that it allows you to enforce policies deterministically, independent of model behavior. If you want to ensure that no file outside the project directory is ever modified, you implement a PreToolUse hook that checks the path argument of every write operation and rejects it if it falls outside the allowed directory. This check runs every single time, regardless of what the model decides. It is not a suggestion to the model; it is a hard constraint enforced by the harness.
Claude Code's sub-agent system is its second distinguishing feature. When a task is too large or complex to handle in a single context window, Claude Code can spawn sub-agents: isolated Claude instances with their own context windows, tool inventories, and permission states. Each sub-agent performs a bounded subtask and returns a final report to the parent agent. This architecture prevents context bloat -- the gradual degradation of model performance as the context window fills with accumulated history -- and enables genuine parallelism. A parent agent might spawn three sub-agents simultaneously: one to research the codebase, one to write the implementation, and one to write the tests. When all three complete, the parent synthesizes their results.
Deep integration with MCP is the third distinguishing feature. Claude Code treats MCP as its primary interface for tool discovery and invocation, which means that any MCP-compatible tool server can be integrated into a Claude Code harness without writing custom integration code.
OPENAI CODEX
OpenAI Codex has had a fascinating evolution. Originally a TypeScript application, it was rewritten in Rust in June 2025, a decision motivated by performance, startup latency, and memory safety. By mid-2026, the codebase is approximately 96 percent Rust. This rewrite produced "Codex Core": a unified agent loop, tool execution engine, and permission management system that powers not just the CLI but also IDE extensions and the ChatGPT desktop application. The fact that the same harness core underlies multiple surfaces is architecturally elegant and reflects a mature understanding of the separation between the harness (a reusable runtime) and the interface (a surface-specific presentation layer).
You install the Codex CLI via npm:
npm install -g @openai/codex
codex --version
Authentication requires setting the OPENAI_API_KEY environment variable, or configuring it in `~/.codex/config.json`. In August 2026, OpenAI reframed Codex as an open-source agent platform under the Apache 2.0 license, explicitly inviting third parties to integrate Codex Core into their own products.
Codex's approach to parallelism is particularly interesting. Rather than using a sub-agent architecture like Claude Code, Codex leverages git worktrees to run multiple agents in parallel on the same codebase. Each agent gets its own worktree -- a separate working directory linked to the same git repository -- and its own branch. Multiple agents can work simultaneously on different features or fixes without interfering with each other's file changes. When each agent completes its work, the results are separate, reviewable diffs that can be merged, rejected, or modified by a human developer.
OPENCODE
OpenCode occupies a different niche. Released under the MIT license and designed to be model-agnostic from the ground up, OpenCode supports more than seventy-five model providers and has become the preferred choice for teams that want the flexibility to switch between Claude Fable 5.1, GPT-6 Astra, Gemini 3.8 Flash, or any of the open-source models like Llama 4 Scout, depending on cost, latency, or capability requirements. Its harness is less opinionated than Claude Code or Codex, which is both a strength and a weakness: it gives teams maximum flexibility but requires more harness engineering work to achieve the same level of reliability.
OPENCLAW
OpenClaw is one of the most interesting stories in the agentic AI landscape and deserves extended treatment. It began as an open-source, self-hosted autonomous agent platform with a distinctive design philosophy: rather than being a coding-specific tool, it was built as a general-purpose agent gateway that could connect any messaging platform -- WhatsApp, Slack, Discord, Telegram, iMessage, and others -- to AI agents. Its creator, Peter Steinberger, built it with the conviction that AI agents should be accessible through the communication channels people already use, rather than requiring them to adopt new interfaces. This vision resonated strongly with the developer community, and OpenClaw experienced viral growth in early 2026.
You install OpenClaw by cloning its repository and building locally:
git clone https://github.com/openclaw/openclaw
cd openclaw
pnpm install
pnpm build
OpenClaw runs as a long-lived Node.js service. For production deployments, Docker is the recommended approach:
docker pull openclaw/openclaw:2.0
docker run -d \
--name openclaw \
-e LLM_API_KEY=your_key_here \
-p 3000:3000 \
openclaw/openclaw:2.0
OpenClaw 2.0, released in mid-2026, introduced major security features including private credential request handling: a mechanism that allows agents to request sensitive credentials from users without exposing them in the conversation history. The platform's ecosystem of more than one hundred "AgentSkills" -- modular capability packages that can be installed into any OpenClaw deployment -- has become one of its most valuable assets, effectively creating a marketplace of harness capabilities.
However, OpenClaw's community edition has significant limitations for enterprise use. It lacks native support for the compliance and auditability requirements of regulated industries: there is no built-in SOC 2 audit trail, no HIPAA-compliant data handling, and no fine-grained role-based access control. Enterprise deployments typically require "hardening" through third-party solutions like ibl.ai's enterprise wrapper or NVIDIA's NemoClaw integration, which adds guardrails, compliance logging, and access control on top of the community edition's core capabilities.
HERMES AGENT (NOUS RESEARCH)
Hermes Agent, developed by Nous Research and released on February 25, 2026 under the MIT license, represents a philosophically rich approach to harness design. Where Claude Code and Codex are primarily coding agents, Hermes is designed as a general-purpose autonomous agent with a particular emphasis on self-improvement and long-term operation.
Installation is straightforward on Linux, macOS, WSL2, or Windows:
curl -fsSL https://hermes-agent.org/install.sh | bash
Alternatively, you can clone and install manually:
git clone https://github.com/nousresearch/hermes-agent
cd hermes-agent
pip install -e .
Hermes Agent requires an LLM API key. You can use any provider via OpenRouter, or a local model via Ollama. Configure your credentials in `~/.hermes/config.yaml`:
llm:
provider: openrouter
api_key: your_openrouter_key_here
model: anthropic/claude-fable-5-1
memory:
backend: sqlite
path: ~/.hermes/memory.db
skills_dir: ~/.hermes/skills/
Start the agent in CLI mode with `hermes` or as an API server with `hermes serve --port 8080`. For messaging platform integration, configure the platform connectors in `~/.hermes/platforms.yaml`, specifying your Slack, Discord, or Telegram credentials.
Hermes Agent's most distinctive feature is what Nous Research calls the closed learning loop: a mechanism by which the agent abstracts successful task completions into reusable skills and refines those skills over time. When Hermes Agent successfully completes a complex task, it can -- either automatically or on command -- distill the approach it used into a SKILL.md document: a structured, slash-command-accessible description of how to perform that type of task. These skills are stored in the agent's skill library and can be retrieved and applied in future sessions. The /learn command automates this process by taking source material -- documentation, code examples, workflow descriptions -- and generating skill documents from it. A Hermes Agent deployment becomes progressively more capable over time as its skill library grows, without requiring any changes to the underlying model.
Hermes Agent supports more than forty built-in tools including web search, browser automation, vision processing, and batch operations. Its MLOps integration is particularly notable: it supports trajectory export in ShareGPT format and integrates with Atropos, Nous Research's reinforcement learning framework, allowing the interaction data generated by Hermes Agent to be used for fine-tuning future model versions. This creates a virtuous cycle in which the agent's operation generates training data that improves the models it runs on.
THE ORCHESTRATION FRAMEWORK LANDSCAPE
LangGraph has emerged as the dominant choice for enterprise production deployments, primarily because of its directed graph-based state machine architecture. Install it with:
pip install langgraph langchain-anthropic
# or for OpenAI models:
pip install langgraph langchain-openai
# verify:
python -c "import langgraph; print('LangGraph ready')"
For production deployments with state persistence, add a PostgreSQL or Redis checkpoint backend:
pip install langgraph-checkpoint-postgres
In a LangGraph harness, the agent's control flow is defined as a graph of nodes (processing steps) and edges (transitions between steps), with typed state objects flowing through the graph. This makes the agent's behavior explicit, auditable, and testable in ways that more dynamic orchestration approaches do not.
CrewAI has found its niche in rapid prototyping and role-based orchestration. Its ergonomics are optimized for teams that want to define agents in terms of roles, goals, and backstories. Install it with `pip install crewai`.
Microsoft Agent Framework 1.0, which reached general availability on April 2, 2026, is the unified successor to both Semantic Kernel and AutoGen (which moved to maintenance mode in October 2025). MAF 1.0 uses YAML for declarative agent definitions, supports graph-based workflow orchestration, and natively implements both MCP and A2A protocols. Install it with `pip install microsoft-agent-framework` for Python or `dotnet add package Microsoft.AgentFramework` for .NET.
Google's Agent Development Kit, operating within the Gemini Enterprise Agent Platform (the rebranded Vertex AI, as of April 2026), introduced `google.adk.Context` in ADK 2.0 for unified tool management and session state isolation. Install with `pip install google-adk`.
Pydantic AI, with its harness-first design introduced in v2.0.0 on June 23, 2026, represents perhaps the most architecturally principled approach in the current ecosystem. Install with `pip install pydantic-ai`. Its central innovation is the "capability" primitive: a single, composable unit that bundles tools, hooks, instructions, and model settings into a cohesive package. The architecture separates the core runtime (which owns the agent loop, message normalization, tool execution, and durable execution primitives) from the harness library (which provides batteries-included compositions like memory, guardrails, sandboxing, sub-agents, and coding tools).
THE PROTOCOL LAYER
What ties all these platforms together is the emergence of shared standards. MCP, donated to the Linux Foundation's Agentic AI Foundation in December 2025, handles the connection between an agent and its tools. The Agent2Agent protocol (A2A), which reached v1.0.0 on March 12, 2026 and is now supported by more than 150 organizations including Google, Microsoft, AWS, and IBM, handles the connection between agents. Where MCP is vertical (agent-to-tool), A2A is horizontal (agent-to-agent). Every A2A-capable agent publishes an Agent Card at `/.well-known/agent-card.json` that advertises its identity, capabilities, and endpoints. A minimal Agent Card looks like this:
{
"schema_version": "1.0.0",
"name": "payments-api-agent",
"description": "Implements GitHub issues for the payments-api service.",
"url": "https://agents.example.com/payments-api",
"capabilities": {
"streaming": true,
"push_notifications": false
},
"skills": [
{
"id": "implement_issue",
"name": "Implement GitHub Issue",
"description": "Reads a GitHub issue and produces a pull request."
}
]
}
An agent that publishes this card can be discovered and delegated to by any other A2A-compatible agent, regardless of which framework built it. This interoperability is still imperfect and requires careful engineering at the integration points, but it represents a qualitative shift from the siloed, incompatible ecosystem of 2024.
CHAPTER 5: DESIGNING A HARNESS -- FROM FIRST PRINCIPLES TO PRODUCTION BLUEPRINT
Design is where theory meets the friction of reality, and where the most consequential engineering decisions are made. A poorly designed harness is not just inefficient -- it is dangerous. It can allow agents to take unauthorized actions, spend unbounded resources, produce outputs that are silently wrong, and fail in ways that are impossible to debug. A well-designed harness, by contrast, is the foundation of a trustworthy agentic system.
The design process for a harness should follow a clear sequence: define the agent's scope and objectives, design the tool layer, design the memory architecture, choose the orchestration pattern, implement the verification loops, define the governance rules, and plan the observability infrastructure. Each of these decisions flows from the previous one, forming a coherent design thread rather than a collection of independent choices.
DEFINING SCOPE AND OBJECTIVES
This step is deceptively important. The most common mistake in harness design is building a harness that is too general: an agent that can do anything, with access to everything, in any order. This kind of agent is hard to test, hard to secure, and hard to trust. The right approach is to start with a clear, specific definition of what the agent is supposed to do and what it is explicitly not supposed to do. This definition drives every subsequent design decision.
Consider a concrete example: you are designing a harness for a software engineering agent whose job is to implement GitHub issues. The scope is clear: the agent reads an issue description, explores the relevant parts of the codebase, writes the implementation, runs the tests, and opens a pull request. It does not deploy to production. It does not modify infrastructure configuration. It does not send emails or post to external services. It does not touch files outside the project repository. Writing down these boundaries explicitly, before designing anything else, is the single most valuable thing you can do at the start of a harness design project.
DESIGNING THE TOOL LAYER
Once the scope is defined, the tool layer design follows naturally. You give the agent exactly the tools it needs to accomplish its defined scope, and no more. For our software engineering agent, the tool set would include file system read and write (scoped to the project directory), shell command execution (scoped to a sandboxed environment with access to the build system and test runner), git operations (read and write, but not push to protected branches), and GitHub API access (to read issues and open pull requests). It would not include web browsing, email sending, or database access to production systems.
The principle at work here is the principle of least privilege, borrowed from traditional security engineering and applied to agent tool design. Every tool you give an agent is a potential failure mode: the agent might use it incorrectly, it might be exploited by an adversary who has injected malicious instructions into the agent's context, or it might have side effects that the agent does not anticipate. The fewer tools the agent has, the smaller the attack surface and the easier the agent is to reason about.
Tool design also requires careful attention to the descriptions and schemas of each tool. The ideal tool description is precise, complete, and written from the model's perspective: what does this tool do, when should I use it, and what should I expect from it? Compare a poorly designed description with a well-designed one for the same file write operation:
Poorly designed: "Writes content to a file."
Well designed: "Writes the given content to a file at the specified path, creating the file if it does not exist and overwriting it if it does. Use this tool when you need to create or modify a source code file, configuration file, or documentation file within the project directory. Do NOT use this tool to write to files outside /workspace/. The path must be an absolute path beginning with /workspace/. Returns 'success' on completion or an error message if the write fails."
The difference is significant. The well-designed description tells the model what the tool does, when to use it, what constraints apply, and what to expect. This reduces incorrect usage and makes the agent's behavior more predictable.
DESIGNING THE MEMORY ARCHITECTURE
The key question is: what does the agent need to remember, for how long, and how does it need to retrieve it? The answers depend on the agent's scope and the nature of its tasks.
For a short-horizon agent -- one that completes tasks in a single session of minutes to hours -- the primary memory concern is managing the context window effectively. The harness needs to ensure that the context window does not overflow with accumulated history, that the most relevant information is always prioritized, and that the model is not confused by stale or contradictory information. Techniques like sliding window conversation history, dynamic summarization of older turns, and selective retrieval of relevant past context are the primary tools here.
For a long-horizon agent -- one that works on tasks spanning days or weeks, or that operates continuously as a persistent service -- the memory architecture is much more complex. The harness needs a persistent storage layer (typically a combination of a relational database for structured state and a vector database for semantic retrieval), a mechanism for deciding what to remember and what to forget, and a reliable retrieval system that can surface relevant memories quickly even as the archive grows large.
Hermes Agent's three-tier memory system -- MEMORY.md for core facts, SQLite FTS5 for episodic recall, and SKILL.md documents for procedural knowledge -- represents a thoughtful balance between simplicity and capability. The MEMORY.md file is small enough to be loaded into every context without consuming significant window space, but it contains the most important facts the agent needs to operate consistently. The SQLite FTS5 index provides fast, full-text search over a much larger archive of past interactions without requiring a vector embedding infrastructure. The SKILL.md documents provide structured, on-demand access to procedural knowledge without cluttering the core context.
CHOOSING THE ORCHESTRATION PATTERN
Many teams get this decision wrong by defaulting to the simplest pattern (sequential ReAct) when their task actually requires something more sophisticated. The right orchestration pattern depends on the nature of the task, the degree of parallelism it admits, the importance of explicit state management, and the tolerance for complexity.
For tasks that are fundamentally sequential and where the steps are well-defined in advance, a pipeline pattern is appropriate and elegant. For tasks that decompose naturally into independent subtasks that can be executed in parallel, a fan-out pattern dramatically reduces latency. For tasks that require dynamic routing -- where the next step depends on the result of the current step in ways that cannot be predicted in advance -- a supervisor pattern with a dynamic planner is the right choice. For tasks where correctness is paramount and the cost of errors is high, a debate pattern that uses multiple agents to cross-check each other's work is worth the additional complexity and cost.
Here is a schematic illustration of how these patterns differ:
Pipeline Pattern:
[Input] -> [Step A] -> [Step B] -> [Step C] -> [Output]
(Sequential, deterministic, low complexity)
Fan-out Pattern:
-> [Agent A] --\
[Input] -> [Orchestrator] -> [Agent B] --> [Synthesizer] -> [Output]
-> [Agent C] --/
(Parallel subtasks, high throughput, requires synthesis step)
Supervisor Pattern:
[Input] -> [Supervisor/Planner]
|
v
[Router] ---> [Worker A] (if task type A)
\---> [Worker B] (if task type B)
\--> [Worker C] (if task type C)
|
[Supervisor/Synthesizer] -> [Output]
(Dynamic routing, highest flexibility, highest complexity)
DESIGNING THE VERIFICATION LOOP
A verification loop has three components: a trigger (what causes verification to run), a checker (what the verification logic actually does), and a handler (what happens when verification fails).
Triggers should be defined for every significant output boundary: every time the agent produces code, every time it proposes to write to a file, every time it makes an API call with side effects, and every time it produces a response that will be shown to a user. In high-stakes environments, triggers should also fire after every tool call, not just at output boundaries.
Checkers can range from simple schema validation (does this JSON have the required fields?) to complex semantic evaluation (does this code actually implement the specified behavior?). Schema validation is cheap and should be applied universally. Semantic evaluation is expensive (it typically requires another model call) and should be reserved for high-stakes outputs.
Handlers define what happens when a check fails. The most common handler is feedback: the failure information is added to the context and the model is asked to try again. This self-repair loop is the most powerful capability of a well-engineered harness. But feedback handlers must be designed carefully to avoid infinite loops: the harness must track the number of repair attempts and escalate to human review if the model cannot produce a passing output after a defined number of tries.
DEFINING GOVERNANCE RULES
Governance rules define the hard constraints that the agent must never violate, regardless of what the model decides. They are typically implemented as a combination of pre-execution checks (is this action permitted?) and post-execution auditing (was this action logged?).
Budget limits deserve special attention because they are the most commonly neglected governance rule and the source of some of the most spectacular production failures in agentic AI. Without a budget ceiling, an agent that gets confused or enters a loop can consume thousands of dollars of API credits in minutes. The harness must track token consumption at every level of granularity -- per model call, per tool execution, per session, per day -- and enforce hard ceilings that terminate the agent's execution if they are exceeded.
PLANNING THE OBSERVABILITY INFRASTRUCTURE
The minimum viable observability infrastructure for a production harness includes structured logging of every agent action (with timestamps, agent identity, action type, inputs, outputs, and outcomes), distributed tracing that correlates related actions across multiple agents and sessions, cost tracking that records token consumption and API call costs at every level of granularity, and alerting that notifies humans when anomalous behavior is detected.
The industry standard for agent tracing is OpenTelemetry-based instrumentation, with traces structured to capture the hierarchical nature of agent execution: a session contains turns, turns contain agent loops, agent loops contain tool calls, and tool calls contain sub-operations. This hierarchical structure allows engineers to navigate from a high-level view of a session down to the specific tool call that caused a problem, without having to wade through an undifferentiated log of events.
CHAPTER 6: IMPLEMENTING A HARNESS -- FROM BLUEPRINT TO RUNNING CODE
Design is theory. Implementation is where theory meets the friction of reality. In this chapter, we work through the implementation of a harness from the ground up, using a concrete example: a software engineering agent that implements GitHub issues for a Python web service. This is a realistic, non-trivial use case that exercises all the major components of a harness.
We use Claude Code as our primary platform, with Pydantic AI providing the capability primitive layer, LangGraph managing orchestration, and MLflow providing observability. This combination represents a state-of-the-art harness stack.
STEP ONE: INSTALL THE STACK
Begin by installing all required packages into a dedicated virtual environment:
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install pydantic-ai
pip install langgraph langgraph-checkpoint-postgres langchain-anthropic
pip install mlflow mlflow-tracing
pip install fastmcp
pip install pytest ruff mypy
pip install python-dotenv
Install Claude Code itself:
curl -fsSL https://claude.ai/install.sh | bash
claude doctor
STEP TWO: SET UP THE INSTRUCTION LAYER
Place this file at the root of your project repository as `./CLAUDE.md`. Claude Code will automatically pick it up at session start because it scans the current directory and all parent directories up to `~/.claude/` for CLAUDE.md files.
PROJECT: payments-api
STACK: Python 3.12, FastAPI 0.115, PostgreSQL 16, pytest 8.3
WORKING CONVENTIONS:
You implement GitHub issues by following these steps in order:
1. Read the issue description carefully and identify the acceptance criteria.
2. Explore the relevant parts of the codebase using read_file and list_files.
3. Identify the files you will need to modify and explain your plan.
4. Implement the changes, following the project conventions below.
5. Run the test suite and fix any failures before proceeding.
6. Open a pull request with a clear description linking to the issue.
CODE CONVENTIONS:
- Use the repository pattern for all database access (src/repositories/).
- Use Pydantic v2 models for all request/response schemas (src/schemas/).
- Follow PEP 8 with 88-character line length (black default).
- All new public functions must have docstrings.
- All new endpoints require integration tests in tests/integration/.
HARD BOUNDARIES (never violate these):
- Do not modify files in migrations/ without explicit confirmation.
- Do not read or write files outside /workspace/payments-api/.
- Do not push to the main or release/* branches.
- Do not make external API calls except to the GitHub API.
- Do not store credentials in any file; use environment variables.
ESCALATION:
If you encounter an architectural ambiguity that requires a design decision,
stop and ask the user before proceeding.
STEP THREE: BUILD THE MCP TOOL SERVER
Create `harness/tool_server.py`. This is a real, runnable MCP server using FastMCP. Run it with `fastmcp run harness/tool_server.py:mcp` before starting your Claude Code session, then register it in your Claude Code settings under MCP servers.
"""
harness/tool_server.py
MCP tool server for the payments-api software engineering agent.
Run with: fastmcp run harness/tool_server.py:mcp
"""
import subprocess
from pathlib import Path
import httpx
from fastmcp import FastMCP
WORKSPACE_ROOT = Path("/workspace/payments-api")
GITHUB_TOKEN = __import__("os").environ.get("GITHUB_TOKEN", "")
GITHUB_REPO = __import__("os").environ.get("GITHUB_REPO", "")
mcp = FastMCP("payments-api-harness")
@mcp.tool
def read_file(path: str) -> str:
"""
Read the contents of a file within the project workspace.
Path must be an absolute path beginning with /workspace/payments-api/.
Returns the file contents as a string, or an error message.
"""
resolved = Path(path).resolve()
if not str(resolved).startswith(str(WORKSPACE_ROOT)):
return f"ERROR: Path '{path}' is outside the allowed workspace."
if not resolved.exists():
return f"ERROR: File '{path}' does not exist."
return resolved.read_text(encoding="utf-8")
@mcp.tool
def write_file(path: str, content: str) -> str:
"""
Write content to a file within the project workspace.
Creates the file if it does not exist; overwrites if it does.
Path must begin with /workspace/payments-api/.
Returns 'success' or an error message.
Do NOT use for files in migrations/ without explicit user confirmation.
"""
resolved = Path(path).resolve()
if not str(resolved).startswith(str(WORKSPACE_ROOT)):
return f"ERROR: Path '{path}' is outside the allowed workspace."
resolved.parent.mkdir(parents=True, exist_ok=True)
resolved.write_text(content, encoding="utf-8")
return "success"
@mcp.tool
def run_tests(test_path: str = "") -> str:
"""
Run the pytest test suite in the project workspace.
Optionally specify a test_path to run a subset of tests.
Returns a structured report including pass/fail counts and
the full output of any failing tests.
"""
cmd = ["python", "-m", "pytest", "--tb=short", "-q"]
if test_path:
cmd.append(test_path)
result = subprocess.run(
cmd,
cwd=str(WORKSPACE_ROOT),
capture_output=True,
text=True,
timeout=300,
)
return result.stdout + result.stderr
@mcp.tool
def run_linter(file_path: str = "") -> str:
"""
Run ruff linter on the specified file or the entire project.
Returns lint errors, or 'No lint errors found.' if clean.
"""
cmd = ["python", "-m", "ruff", "check"]
if file_path:
cmd.append(file_path)
else:
cmd.append(str(WORKSPACE_ROOT / "src"))
result = subprocess.run(
cmd,
cwd=str(WORKSPACE_ROOT),
capture_output=True,
text=True,
)
return result.stdout or "No lint errors found."
@mcp.tool
def open_pull_request(title: str, body: str, issue_number: int) -> str:
"""
Open a GitHub pull request from the current branch to main.
Links the pull request to the specified issue number.
Requires GITHUB_TOKEN and GITHUB_REPO environment variables.
Returns the pull request URL on success or an error message.
"""
branch_result = subprocess.run(
["git", "rev-parse", "--abbrev-ref", "HEAD"],
cwd=str(WORKSPACE_ROOT),
capture_output=True,
text=True,
)
current_branch = branch_result.stdout.strip()
if current_branch in ("main", "master") or current_branch.startswith("release/"):
return "ERROR: Cannot open pull request from a protected branch."
url = f"https://api.github.com/repos/{GITHUB_REPO}/pulls"
payload = {
"title": title,
"body": f"{body}\n\nCloses #{issue_number}",
"head": current_branch,
"base": "main",
}
response = httpx.post(
url,
json=payload,
headers={"Authorization": f"Bearer {GITHUB_TOKEN}"},
timeout=30,
)
if response.status_code == 201:
return response.json()["html_url"]
return f"ERROR: GitHub API returned {response.status_code}: {response.text}"
STEP FOUR: IMPLEMENT THE PRETOOLUSE HOOK
Create `harness/hooks.py`. This module provides the PreToolUse hook that Claude Code will call before every tool execution. Register it in your Claude Code session settings as a hook handler.
"""
harness/hooks.py
PreToolUse hook for the payments-api agent harness.
Enforces path boundaries, protected-path rules, and audit logging.
"""
import json
import logging
from datetime import datetime, timezone
from pathlib import Path
ALLOWED_PATH_PREFIX = "/workspace/payments-api/"
PROTECTED_WRITE_PATHS = ["/workspace/payments-api/migrations/"]
PROTECTED_BRANCHES = ["main", "master", "release/"]
# Configure a structured audit logger that writes to a dedicated file.
audit_logger = logging.getLogger("harness.audit")
audit_logger.setLevel(logging.INFO)
_handler = logging.FileHandler("/workspace/payments-api/.harness/audit.log")
_handler.setFormatter(logging.Formatter("%(message)s"))
audit_logger.addHandler(_handler)
def _audit(tool_name: str, tool_input: dict, approved: bool, reason: str = "") -> None:
"""Write a structured audit record for every tool call attempt."""
record = {
"timestamp": datetime.now(timezone.utc).isoformat(),
"tool": tool_name,
"input": tool_input,
"approved": approved,
"reason": reason,
}
audit_logger.info(json.dumps(record))
def pre_tool_use_hook(tool_name: str, tool_input: dict) -> dict:
"""
Validates every tool call before execution.
Returns {"approved": True} or {"approved": False, "reason": "..."}.
Claude Code calls this function automatically via its hook system.
"""
# Enforce path boundaries for file read/write operations.
if tool_name in ("read_file", "write_file"):
path = tool_input.get("path", "")
if not path.startswith(ALLOWED_PATH_PREFIX):
reason = (
f"Path '{path}' is outside the allowed workspace. "
f"All file operations must be within {ALLOWED_PATH_PREFIX}."
)
_audit(tool_name, tool_input, approved=False, reason=reason)
return {"approved": False, "reason": reason}
# Writes to protected paths require explicit user confirmation.
if tool_name == "write_file":
for protected in PROTECTED_WRITE_PATHS:
if path.startswith(protected):
reason = (
f"Writes to '{protected}' require explicit user "
f"confirmation. Please ask the user before proceeding."
)
_audit(tool_name, tool_input, approved=False, reason=reason)
return {"approved": False, "reason": reason}
# All checks passed. Log the approved call and allow execution.
_audit(tool_name, tool_input, approved=True)
return {"approved": True}
STEP FIVE: IMPLEMENT THE VERIFICATION LOOP
Create `harness/verification.py`. This module runs after the agent has written code and before it proceeds to the next step.
"""
harness/verification.py
Verification loop for the payments-api agent harness.
Runs linting, type checking, and tests; feeds failures back to the agent;
escalates to human review if the maximum repair attempts are exceeded.
"""
import subprocess
from dataclasses import dataclass, field
from pathlib import Path
WORKSPACE_ROOT = Path("/workspace/payments-api")
MAX_REPAIR_ATTEMPTS = 3
@dataclass
class VerificationResult:
"""Captures the outcome of a full verification pass."""
success: bool
attempts: int
escalation_required: bool = False
escalation_message: str = ""
feedback_history: list[str] = field(default_factory=list)
def _run_ruff(target: str = "") -> tuple[bool, str]:
"""Run ruff linter. Returns (passed, output)."""
cmd = ["python", "-m", "ruff", "check", target or str(WORKSPACE_ROOT / "src")]
result = subprocess.run(
cmd, cwd=str(WORKSPACE_ROOT), capture_output=True, text=True
)
passed = result.returncode == 0
return passed, result.stdout + result.stderr
def _run_mypy(target: str = "") -> tuple[bool, str]:
"""Run mypy type checker. Returns (passed, output)."""
cmd = ["python", "-m", "mypy", target or str(WORKSPACE_ROOT / "src")]
result = subprocess.run(
cmd, cwd=str(WORKSPACE_ROOT), capture_output=True, text=True
)
passed = result.returncode == 0
return passed, result.stdout + result.stderr
def _run_pytest(scope: str = "tests/integration/") -> tuple[bool, str]:
"""Run pytest. Returns (passed, output)."""
cmd = ["python", "-m", "pytest", scope, "--tb=short", "-q"]
result = subprocess.run(
cmd, cwd=str(WORKSPACE_ROOT), capture_output=True, text=True, timeout=300
)
passed = result.returncode == 0
return passed, result.stdout + result.stderr
def verify_and_repair(agent, generated_files: list[str]) -> VerificationResult:
"""
Run linting, type checking, and tests on generated files.
Feed failures back to the agent for repair.
Escalate to human review if MAX_REPAIR_ATTEMPTS is exceeded.
Parameters
----------
agent : The running agent instance with an add_feedback() method.
generated_files : List of file paths the agent has written.
"""
feedback_history: list[str] = []
for attempt in range(1, MAX_REPAIR_ATTEMPTS + 1):
lint_passed, lint_output = _run_ruff()
type_passed, type_output = _run_mypy()
test_passed, test_output = _run_pytest()
if lint_passed and type_passed and test_passed:
return VerificationResult(
success=True,
attempts=attempt,
feedback_history=feedback_history,
)
# Build structured feedback for the model.
feedback_parts = []
if not lint_passed:
feedback_parts.append(f"Lint errors:\n{lint_output}")
if not type_passed:
feedback_parts.append(f"Type errors:\n{type_output}")
if not test_passed:
feedback_parts.append(f"Test failures:\n{test_output}")
feedback = "\n\n".join(feedback_parts)
feedback_history.append(feedback)
# Feed back to the agent for repair.
agent.add_feedback(
f"Verification attempt {attempt} failed. "
f"Please fix the following issues:\n\n{feedback}"
)
# Maximum attempts exceeded: escalate to human.
escalation_message = (
f"Agent could not produce passing code after {MAX_REPAIR_ATTEMPTS} "
f"attempts. Human review is required. "
f"Final feedback:\n\n{feedback_history[-1]}"
)
return VerificationResult(
success=False,
attempts=MAX_REPAIR_ATTEMPTS,
escalation_required=True,
escalation_message=escalation_message,
feedback_history=feedback_history,
)
STEP SIX: IMPLEMENT THE BUDGET ENFORCER
Create `harness/budget.py`. This module tracks token consumption and cost across the session and terminates execution if limits are exceeded.
"""
harness/budget.py
Budget enforcement for the payments-api agent harness.
Tracks token consumption and cost; raises BudgetExceededError if limits
are exceeded. Wrap every model call with budget.record_model_call().
"""
from datetime import datetime, timezone
from dataclasses import dataclass, field
# Cost per million tokens (input/output) for each model as of Sep 2026.
# Update these values when model pricing changes.
MODEL_PRICING: dict[str, dict[str, float]] = {
"claude-fable-5-1": {"input": 3.00, "output": 15.00},
"claude-fable-5": {"input": 3.00, "output": 15.00},
"gpt-6-astra": {"input": 5.00, "output": 20.00},
"gemini-3-8-flash": {"input": 0.075, "output": 0.30},
}
def _calculate_cost(model: str, input_tokens: int, output_tokens: int) -> float:
"""Return the USD cost for a single model call."""
pricing = MODEL_PRICING.get(model, {"input": 5.00, "output": 20.00})
return (
input_tokens * pricing["input"] / 1_000_000
+ output_tokens * pricing["output"] / 1_000_000
)
class BudgetExceededError(RuntimeError):
"""Raised when any budget limit (tokens, cost, or time) is exceeded."""
@dataclass
class BudgetEnforcer:
"""
Tracks and enforces token, cost, and time budgets for an agent session.
Usage:
budget = BudgetEnforcer(max_tokens=500_000, max_cost_usd=5.00,
max_duration_minutes=30)
# After each model call:
budget.record_model_call(
input_tokens=resp.usage.input_tokens,
output_tokens=resp.usage.output_tokens,
model="claude-fable-5-1",
)
"""
max_tokens: int
max_cost_usd: float
max_duration_minutes: int
tokens_used: int = field(default=0, init=False)
cost_usd: float = field(default=0.0, init=False)
start_time: datetime = field(
default_factory=lambda: datetime.now(timezone.utc), init=False
)
call_log: list[dict] = field(default_factory=list, init=False)
def record_model_call(
self, input_tokens: int, output_tokens: int, model: str
) -> None:
"""Record a model call and check all budget limits."""
call_cost = _calculate_cost(model, input_tokens, output_tokens)
self.tokens_used += input_tokens + output_tokens
self.cost_usd += call_cost
self.call_log.append(
{
"model": model,
"input_tokens": input_tokens,
"output_tokens": output_tokens,
"cost_usd": call_cost,
}
)
self._check_limits()
def _check_limits(self) -> None:
elapsed_minutes = (
datetime.now(timezone.utc) - self.start_time
).total_seconds() / 60
if self.tokens_used > self.max_tokens:
raise BudgetExceededError(
f"Token limit exceeded: {self.tokens_used:,} > {self.max_tokens:,}"
)
if self.cost_usd > self.max_cost_usd:
raise BudgetExceededError(
f"Cost limit exceeded: ${self.cost_usd:.4f} > ${self.max_cost_usd:.2f}"
)
if elapsed_minutes > self.max_duration_minutes:
raise BudgetExceededError(
f"Time limit exceeded: {elapsed_minutes:.1f}m "
f"> {self.max_duration_minutes}m"
)
def summary(self) -> dict:
"""Return a summary of current budget consumption."""
elapsed = (
datetime.now(timezone.utc) - self.start_time
).total_seconds() / 60
return {
"tokens_used": self.tokens_used,
"tokens_remaining": self.max_tokens - self.tokens_used,
"cost_usd": round(self.cost_usd, 4),
"cost_remaining_usd": round(self.max_cost_usd - self.cost_usd, 4),
"elapsed_minutes": round(elapsed, 1),
"call_count": len(self.call_log),
}
STEP SEVEN: WIRE UP THE OBSERVABILITY LAYER
Create `harness/observability.py`. This module instruments the agent session with MLflow tracking and OpenTelemetry tracing.
"""
harness/observability.py
MLflow + OpenTelemetry observability for the payments-api agent harness.
Setup:
export MLFLOW_TRACKING_URI=http://localhost:5000
mlflow server --host 0.0.0.0 --port 5000 # start tracking server
python -m harness.observability # verify connection
Usage:
from harness.observability import traced_agent_session
result = traced_agent_session(issue_number=42,
issue_description="Add rate limiting...")
"""
import mlflow
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from harness.budget import BudgetEnforcer, BudgetExceededError
# Configure OpenTelemetry tracing.
_provider = TracerProvider()
_provider.add_span_processor(
BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317"))
)
trace.set_tracer_provider(_provider)
_tracer = trace.get_tracer("payments-api-harness")
# Configure MLflow experiment.
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("payments-api-agent")
def traced_agent_session(
issue_number: int,
issue_description: str,
model: str = "claude-fable-5-1",
max_tokens: int = 500_000,
max_cost_usd: float = 5.00,
max_duration_minutes: int = 30,
) -> dict:
"""
Execute a full agent session with MLflow tracking and OTel tracing.
Returns a summary dict with success status, cost, tokens, and PR URL.
"""
budget = BudgetEnforcer(
max_tokens=max_tokens,
max_cost_usd=max_cost_usd,
max_duration_minutes=max_duration_minutes,
)
with mlflow.start_run(run_name=f"issue-{issue_number}") as run:
mlflow.log_params(
{
"issue_number": issue_number,
"model": model,
"max_tokens": max_tokens,
"max_cost_usd": max_cost_usd,
}
)
with _tracer.start_as_current_span("agent_session") as session_span:
session_span.set_attribute("issue.number", issue_number)
session_span.set_attribute("agent.model", model)
try:
# Import and run the agent. The agent calls budget.record_model_call()
# after each LLM invocation and raises BudgetExceededError if limits
# are breached.
from harness.agent import run_agent # noqa: PLC0415
result = run_agent(
issue_number=issue_number,
issue_description=issue_description,
budget=budget,
)
summary = budget.summary()
mlflow.log_metrics(
{
"success": 1,
"tokens_used": summary["tokens_used"],
"cost_usd": summary["cost_usd"],
"elapsed_minutes": summary["elapsed_minutes"],
"repair_attempts": result.get("repair_attempts", 0),
}
)
session_span.set_attribute("success", True)
return {"success": True, "pr_url": result.get("pr_url"), **summary}
except BudgetExceededError as exc:
summary = budget.summary()
mlflow.log_metrics({"success": 0, **summary})
mlflow.log_param("failure_reason", "budget_exceeded")
mlflow.log_text(str(exc), "error.txt")
session_span.set_attribute("success", False)
session_span.record_exception(exc)
return {"success": False, "failure_reason": str(exc), **summary}
except Exception as exc:
summary = budget.summary()
mlflow.log_metrics({"success": 0, **summary})
mlflow.log_param("failure_reason", "agent_error")
mlflow.log_text(str(exc), "error.txt")
session_span.set_attribute("success", False)
session_span.record_exception(exc)
raise
STEP EIGHT: RUN THE COMPLETE STACK
With all files in place, start the supporting services and then launch your agent session:
# Terminal 1: start the MLflow tracking server
mlflow server --host 0.0.0.0 --port 5000
# Terminal 2: start the MCP tool server
fastmcp run harness/tool_server.py:mcp
# Terminal 3: run an agent session against a specific GitHub issue
cd /workspace/payments-api
python -c "
from harness.observability import traced_agent_session
result = traced_agent_session(
issue_number=142,
issue_description='Add rate limiting middleware to all API endpoints.',
)
print(result)
Alternatively, start an interactive Claude Code session in the project directory and let it read the CLAUDE.md file automatically:
cd /workspace/payments-api
claude
Within the Claude Code session, the MCP tool server you started in Terminal 2 will be available as a registered tool provider. The PreToolUse hook in `harness/hooks.py` must be registered in your Claude Code settings file at `~/.claude/settings.json`:
{
"hooks": {
"PreToolUse": [
{
"type": "python",
"module": "harness.hooks",
"function": "pre_tool_use_hook"
}
]
},
"mcp_servers": {
"payments-api-harness": {
"command": "fastmcp",
"args": ["run", "harness/tool_server.py:mcp"]
}
}
}
This implementation gives you a complete, production-grade harness. It is not a toy: it enforces hard boundaries, verifies outputs, controls costs, and produces the structured observability data you need to understand and improve the agent's behavior over time.
One implementation detail that deserves special mention is the handling of the human-in-the-loop escalation path. In the code above, escalation is handled by returning an escalation_required flag and message. In a production system, the escalation mechanism should pause the agent's execution in a resumable state (rather than terminating it), notify the appropriate human via a configured channel (Slack, email, PagerDuty, or a custom dashboard), provide the human with enough context to make an informed decision, and resume the agent's execution with the human's decision incorporated into the context.
LangGraph's native checkpointing capability is particularly valuable here. When an agent node reaches an escalation condition, it can checkpoint its current state to persistent storage and pause. When a human provides input, the harness loads the checkpoint, adds the human's input to the context, and resumes execution from the exact point where it paused. This pause-and-resume capability is essential for long-horizon agents that may encounter multiple escalation points over the course of a complex task.
CHAPTER 7: SECURITY IN HARNESS ENGINEERING -- DEFENDING THE AGENT FROM ITSELF AND FROM ADVERSARIES
Security is not a feature you add to a harness after it is built. It is a design constraint that shapes every decision from the earliest stages of architecture. This is a lesson the industry has learned painfully, as the 2026 security landscape for agentic AI makes clear: 88.4 percent of organizations reported agent-related security incidents in the twelve months preceding mid-2026. The attack surfaces are novel, the consequences of failures are severe, and the traditional security playbook does not map cleanly onto the unique threat model of autonomous agents.
PROMPT INJECTION
The most pervasive and dangerous threat is prompt injection. In a traditional web application, injection attacks exploit the failure to distinguish between code and data: a SQL injection attack tricks a database into treating user-supplied data as SQL commands. Prompt injection exploits an analogous failure in language models: the model's inability to reliably distinguish between its instructions (which it should follow) and data it is processing (which it should treat as inert content). An attacker who can get malicious instructions into the model's context -- through a poisoned document, a manipulated tool output, or a crafted user message -- can potentially redirect the agent's behavior in arbitrary ways.
Prompt injection attacks against agents are more dangerous than against simple chatbots because agents have tools. A chatbot that is successfully injected can produce harmful text. An agent that is successfully injected can execute arbitrary actions: exfiltrate files, delete data, make unauthorized API calls, or manipulate its own memory to persist the attack across sessions.
The harness-level defenses against prompt injection operate at multiple layers. Structural separation is the instruction layer defense: the agent's system instructions should be delivered through a channel that is architecturally distinct from the data channels. In Claude Code, the CLAUDE.md file and the system prompt are loaded through a privileged channel that the model is trained to treat as authoritative; tool outputs and retrieved documents are delivered through a separate channel that the model is trained to treat as potentially untrusted data. This structural separation does not make injection impossible, but it raises the bar significantly.
Input validation and output sanitization are the tool layer defenses. Every tool input should be validated against a strict schema before execution, and every tool output should be sanitized before being fed back into the model's context. The sanitization step is particularly important for tools that retrieve content from external sources: a web search result, a retrieved document, or an API response might contain adversarially crafted content designed to inject instructions. The harness should strip or escape any content that looks like instructions before adding it to the context.
Semantic consistency checking is the verification layer defense. After every significant agent action, the harness can run a quick consistency check: does the agent's current plan still align with its original objective? If the agent was supposed to implement a GitHub issue and is now attempting to exfiltrate the contents of the credentials directory, something has gone wrong. This kind of behavioral anomaly detection is not foolproof, but it catches a wide range of injection attacks that manage to get past the structural defenses.
TOOL POISONING
Tool poisoning is a specific form of supply-chain attack targeting the MCP ecosystem. The attack works by embedding adversarial instructions in the description or metadata of an MCP tool server. When the agent loads the tool definitions, it reads these descriptions to understand how to use the tools. If the descriptions contain hidden instructions -- written in a way that is invisible to human reviewers but visible to the model -- the agent may follow those instructions in ways the human operator did not intend.
Tool poisoning is particularly concerning because it can be subtle and persistent. An attacker who compromises a widely-used MCP server can potentially influence the behavior of every agent that connects to it. The harness-level defenses include vetting every MCP server before connecting to it (using the same due diligence you would apply to any third-party library), pinning tool server versions to prevent silent updates, and using an agent registry that tracks the provenance, version, and integrity of every tool server in use. Enterprise agent registries have become essential infrastructure for organizations that operate at scale, serving the same function for tool servers that software composition analysis tools serve for library dependencies.
EXCESSIVE AGENCY
Excessive agency is the tendency of agents, when given broad permissions and ambitious goals, to take actions that are technically within their permissions but outside the spirit of what was intended. This is not an adversarial attack -- it is an emergent property of capable, goal-directed systems operating in complex environments. An agent tasked with "improve the performance of the test suite" might decide that the most efficient way to do this is to delete the slow tests. Technically, this reduces the test suite's runtime. Practically, it is a disaster.
The harness-level defenses against excessive agency are the governance rules discussed in the design chapter, combined with a careful application of the principle of least privilege. The agent should have the minimum permissions necessary to accomplish its defined scope, and the verification layer should check not just whether the agent's outputs are technically correct but whether they align with the intended objective. This is one of the areas where human-in-the-loop escalation is most valuable: for actions that are technically permitted but that have large, hard-to-reverse consequences, the harness should require human confirmation before proceeding.
MEMORY POISONING
Long-horizon agents that maintain persistent memory across sessions are vulnerable to attacks that corrupt or manipulate their memory stores. An attacker who can inject false information into an agent's MEMORY.md or episodic memory database can influence the agent's behavior in future sessions, potentially in ways that are very difficult to detect. The harness-level defenses include integrity checking of memory stores (using cryptographic signatures to detect unauthorized modifications), provenance tracking (recording the source of every memory entry so that suspicious entries can be identified and removed), and periodic memory audits that review the agent's stored knowledge for anomalies.
The OWASP Top 10 for AI Agents, which has become the industry's primary framework for categorizing agentic AI security risks, identifies memory poisoning, excessive agency, prompt injection, tool poisoning, and goal hijacking as the five most critical risk categories. The OWASP framework also provides specific mitigation recommendations for each category, and any serious harness engineering project should use it as a checklist.
NON-HUMAN IDENTITY MANAGEMENT
When an agent takes an action -- writing a file, calling an API, opening a pull request -- who is responsible for that action? The agent is not a human employee, but it is not a simple automated script either. It makes decisions, it has goals, and it can cause real-world consequences. Organizations need to treat agents as distinct identities with their own access controls, audit trails, and accountability mechanisms.
In practice, this means giving each agent deployment a dedicated service account with scoped permissions, rather than running agents under a human user's credentials or under a broad service account shared with other systems. It means logging every agent action in a way that preserves the agent's identity, the session in which the action occurred, and the chain of reasoning that led to the action. Short-lived, scoped credentials are the recommended practice: rather than giving an agent a long-lived API key that provides broad access, the harness should provision a fresh, scoped credential at the start of each session and revoke it when the session ends. This limits the blast radius of a credential compromise and makes it much harder for an attacker to use a stolen credential to cause persistent harm.
SANDBOX DESIGN
Many agent harnesses give agents access to code execution environments -- shell access, Python interpreters, container runtimes -- that are inherently powerful and potentially dangerous. The harness must ensure that these execution environments are properly sandboxed: isolated from the host system, from other agents, and from sensitive data that the agent should not have access to.
The standard approach to agent sandboxing is container-based isolation: each agent session runs in a dedicated container with a minimal filesystem, no network access by default (with specific external endpoints whitelisted as needed), and resource limits on CPU, memory, and disk usage. The container is destroyed at the end of the session, ensuring that any state the agent created during the session does not persist in ways that could affect future sessions or other users. For particularly sensitive environments, hardware-level isolation using technologies like Firecracker microVMs provides an additional layer of protection.
CHAPTER 8: TESTING A HARNESS -- BECAUSE HOPE IS NOT A STRATEGY
Testing agentic systems is genuinely hard. The non-determinism of language models means that the same harness, given the same input, may produce different outputs on different runs. The long-horizon nature of many agent tasks means that failures may not manifest until dozens of steps into an execution. The complex interactions between the model, the tools, the memory, and the orchestration logic mean that failures often emerge from the interaction of components that each work correctly in isolation. And the high cost of running frontier models means that exhaustive testing is economically prohibitive.
These challenges are real, but they are not insurmountable. The key insight is that testing an agentic harness requires a different methodology than testing a traditional software system, one that is adapted to the non-deterministic, multi-step, expensive nature of agent execution.
UNIT TESTING HARNESS COMPONENTS
While the full agent loop is hard to test in isolation, the individual components of the harness are not. The PreToolUse hooks, the verification logic, the budget enforcement, the memory retrieval, and the orchestration routing can all be tested with traditional unit tests that do not require model calls. These tests are fast, cheap, and deterministic, and they should be the foundation of the harness test suite.
Here is a complete unit test suite for the PreToolUse hook implemented in Chapter 6. Run it with `pytest harness/tests/test_hooks.py -v`.
"""
harness/tests/test_hooks.py
Unit tests for the PreToolUse hook.
Run with: pytest harness/tests/test_hooks.py -v
No model calls required; all tests run in milliseconds.
"""
import pytest
from harness.hooks import pre_tool_use_hook
class TestPathBoundaryEnforcement:
"""Verify that file operations outside the workspace are rejected."""
def test_rejects_write_to_system_path(self):
result = pre_tool_use_hook(
tool_name="write_file",
tool_input={"path": "/etc/passwd", "content": "hacked"},
)
assert result["approved"] is False
assert "outside the allowed workspace" in result["reason"]
def test_rejects_read_from_home_directory(self):
result = pre_tool_use_hook(
tool_name="read_file",
tool_input={"path": "/home/user/.ssh/id_rsa"},
)
assert result["approved"] is False
assert "outside the allowed workspace" in result["reason"]
def test_approves_valid_write_path(self):
result = pre_tool_use_hook(
tool_name="write_file",
tool_input={
"path": "/workspace/payments-api/src/api/routes.py",
"content": "# new code",
},
)
assert result["approved"] is True
def test_approves_valid_read_path(self):
result = pre_tool_use_hook(
tool_name="read_file",
tool_input={"path": "/workspace/payments-api/src/api/routes.py"},
)
assert result["approved"] is True
class TestProtectedPathEnforcement:
"""Verify that writes to protected paths are blocked pending confirmation."""
def test_rejects_write_to_migrations(self):
result = pre_tool_use_hook(
tool_name="write_file",
tool_input={
"path": "/workspace/payments-api/migrations/002_add_index.sql",
"content": "DROP TABLE users;",
},
)
assert result["approved"] is False
assert "explicit user confirmation" in result["reason"]
def test_allows_read_from_migrations(self):
"""Reads from protected paths should be allowed; only writes are blocked."""
result = pre_tool_use_hook(
tool_name="read_file",
tool_input={
"path": "/workspace/payments-api/migrations/001_init.sql"
},
)
assert result["approved"] is True
class TestNonFileTools:
"""Verify that non-file tools pass through without path checks."""
def test_approves_run_tests(self):
result = pre_tool_use_hook(
tool_name="run_tests",
tool_input={"test_path": "tests/integration/"},
)
assert result["approved"] is True
def test_approves_run_linter(self):
result = pre_tool_use_hook(
tool_name="run_linter",
tool_input={},
)
assert result["approved"] is True
These tests run in milliseconds, require no model calls, and provide high confidence that the governance rules are correctly implemented. They should run in CI on every commit to the harness codebase.
INTEGRATION TESTING WITH MOCK MODELS
For testing the interaction between harness components -- how the orchestration layer responds to different model outputs, how the verification loop handles different types of failures, how the memory system retrieves relevant context -- it is valuable to use mock models that return predetermined outputs rather than real model calls. This allows you to test complex multi-step scenarios deterministically and cheaply.
"""
harness/tests/test_verification_loop.py
Integration tests for the verification loop using a mock model.
Tests the full repair cycle without making real LLM API calls.
Run with: pytest harness/tests/test_verification_loop.py -v
"""
import pytest
from dataclasses import dataclass, field
from harness.verification import verify_and_repair, VerificationResult
@dataclass
class MockAgent:
"""
A deterministic mock agent for harness integration testing.
Records all feedback messages it receives from the verification loop.
"""
feedback_received: list[str] = field(default_factory=list)
def add_feedback(self, message: str) -> None:
self.feedback_received.append(message)
class TestVerificationLoopBehavior:
"""Verify the verification loop's repair and escalation logic."""
def test_records_feedback_on_failure(self, tmp_path, monkeypatch):
"""
When linting fails, the loop must call agent.add_feedback()
with a message containing the lint errors.
"""
# Patch _run_ruff to simulate a lint failure on attempt 1,
# then a pass on attempt 2.
call_count = {"n": 0}
def mock_ruff(*args, **kwargs):
call_count["n"] += 1
if call_count["n"] == 1:
return False, "E501 line too long (92 > 88 characters)"
return True, ""
def mock_mypy(*args, **kwargs):
return True, ""
def mock_pytest(*args, **kwargs):
return True, ""
monkeypatch.setattr("harness.verification._run_ruff", mock_ruff)
monkeypatch.setattr("harness.verification._run_mypy", mock_mypy)
monkeypatch.setattr("harness.verification._run_pytest", mock_pytest)
agent = MockAgent()
result = verify_and_repair(agent, generated_files=[])
assert result.success is True
assert result.attempts == 2
assert len(agent.feedback_received) == 1
assert "Lint errors" in agent.feedback_received[0]
def test_escalates_after_max_attempts(self, monkeypatch):
"""
When all repair attempts fail, the loop must set
escalation_required=True and provide an escalation message.
"""
def always_fail_ruff(*args, **kwargs):
return False, "persistent lint error"
def always_pass(*args, **kwargs):
return True, ""
monkeypatch.setattr("harness.verification._run_ruff", always_fail_ruff)
monkeypatch.setattr("harness.verification._run_mypy", always_pass)
monkeypatch.setattr("harness.verification._run_pytest", always_pass)
agent = MockAgent()
result = verify_and_repair(agent, generated_files=[])
assert result.success is False
assert result.escalation_required is True
assert "Human review" in result.escalation_message
assert result.attempts == 3
TRACE-BASED EVALUATION
Trace-based evaluation is the most powerful and most distinctively agentic testing technique. Rather than testing individual components in isolation, it tests the agent's end-to-end behavior on a set of representative tasks, recording the full execution trace of each run and evaluating it against a set of criteria.
Set up an MLflow-based evaluation run with a dataset of representative tasks:
"""
harness/tests/eval_traces.py
Trace-based evaluation using MLflow genai.evaluate().
Run with: python harness/tests/eval_traces.py
Requires: MLFLOW_TRACKING_URI set to your tracking server.
"""
import mlflow
import mlflow.genai
# Define your evaluation dataset: a list of representative tasks
# with expected outcomes. Build this from real production examples.
EVAL_DATASET = [
{
"inputs": {
"issue_number": 1,
"issue_description": "Add a /health endpoint that returns 200 OK.",
},
"expected_outputs": {
"success": True,
"files_modified": ["src/api/routes.py", "tests/integration/test_health.py"],
},
},
{
"inputs": {
"issue_number": 2,
"issue_description": "Add input validation to the /payments POST endpoint.",
},
"expected_outputs": {
"success": True,
"files_modified": ["src/api/routes.py", "src/schemas/payment.py"],
},
},
]
def task_completion_scorer(run, expected_output) -> mlflow.genai.Score:
"""Custom scorer: did the agent complete the task successfully?"""
actual_success = run.outputs.get("success", False)
expected_success = expected_output.get("success", True)
return mlflow.genai.Score(
name="task_completion",
value=1.0 if actual_success == expected_success else 0.0,
justification="Agent completed the task" if actual_success else "Task failed",
)
def cost_efficiency_scorer(run, expected_output) -> mlflow.genai.Score:
"""Custom scorer: did the agent stay within a reasonable cost budget?"""
cost = run.outputs.get("cost_usd", 999)
score = 1.0 if cost < 1.00 else (0.5 if cost < 3.00 else 0.0)
return mlflow.genai.Score(
name="cost_efficiency",
value=score,
justification=f"Session cost: ${cost:.4f}",
)
if __name__ == "__main__":
mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("payments-api-agent-eval")
results = mlflow.genai.evaluate(
data=EVAL_DATASET,
scorers=[task_completion_scorer, cost_efficiency_scorer],
# The predict_fn wraps your agent session for evaluation.
predict_fn=lambda inputs: __import__(
"harness.observability", fromlist=["traced_agent_session"]
).traced_agent_session(**inputs),
)
print(f"Mean task completion: {results.metrics['mean/task_completion']:.2%}")
print(f"Mean cost efficiency: {results.metrics['mean/cost_efficiency']:.2%}")
ADVERSARIAL TESTING
Red-teaming for agentic harnesses should cover at minimum the following scenarios. Direct prompt injection means the user directly instructs the agent to ignore its system prompt and do something it should not. Indirect prompt injection means a document or tool output that the agent retrieves contains hidden instructions. Tool poisoning means a tool's description has been modified to contain adversarial instructions. Memory poisoning means the agent's persistent memory has been corrupted with false information. Budget exhaustion means the user or an adversary attempts to exhaust the agent's budget by submitting a large number of complex tasks or by crafting a task that causes the agent to loop. Privilege escalation means the agent attempts to acquire permissions it was not given, either through creative tool use or by convincing a human to grant additional access.
A practical starting point for red-teaming is to craft a set of adversarial test inputs and run them against your harness, verifying that the governance controls catch each one:
"""
harness/tests/test_adversarial.py
Adversarial tests for the payments-api agent harness.
These tests verify that security controls catch common attack patterns.
Run with: pytest harness/tests/test_adversarial.py -v
"""
import pytest
from harness.hooks import pre_tool_use_hook
class TestPromptInjectionViaPath:
"""
Verify that path-based injection attempts are caught by the hook layer.
These tests simulate an adversary who has manipulated the agent's
tool input to target files outside the workspace.
"""
INJECTION_PATHS = [
"/etc/shadow",
"/root/.ssh/authorized_keys",
"../../.env",
"/workspace/payments-api/../../../etc/cron.d/backdoor",
]
@pytest.mark.parametrize("path", INJECTION_PATHS)
def test_rejects_path_traversal_attempt(self, path):
result = pre_tool_use_hook(
tool_name="write_file",
tool_input={"path": path, "content": "malicious content"},
)
assert result["approved"] is False, (
f"Path traversal attempt was not caught: {path}"
)
PERFORMANCE TESTING
A particularly important performance test for long-horizon agents is the "fifty tool calls" test: run the agent on a task that requires more than fifty sequential tool calls and observe whether its performance degrades. Research has shown that many agents exhibit significant performance degradation on long-horizon tasks due to attention dilution (the model's tendency to pay less attention to information that is far back in the context window) and accumulated error propagation (small errors early in the task that compound over many steps). A well-designed harness mitigates these effects through context compaction, periodic summarization, and verification checkpoints.
The testing discipline for harnesses is, in summary, a multi-layered approach that combines the fast, cheap, deterministic testing of traditional software engineering with the statistical, trace-based evaluation methods that are unique to agentic systems. Unit tests alone cannot catch emergent failures in the full agent loop, and trace-based evaluation alone is too slow and expensive to run on every commit. The combination of both, organized into a coherent test pyramid with appropriate coverage at each level, is what production-grade harness testing looks like.
CHAPTER 9: THE FUTURE OF HARNESS ENGINEERING -- WHERE THIS IS ALL GOING
It would be intellectually dishonest to write a comprehensive essay on Harness Engineering without acknowledging that the discipline is moving very fast and that some of what is true today will be obsolete in twelve months. The history of this field, as we traced it in Chapter 2, is a history of rapid paradigm shifts: prompt engineering to context engineering to harness engineering in roughly four years. The next shift is already visible on the horizon.
NATIVELY AGENTIC MODELS
The frontier models -- Claude Fable 5.1, GPT-6 Astra, Gemini 3.1 Pro -- were trained with agentic use cases in mind, but they are still fundamentally language models that have been adapted for tool use through instruction tuning and reinforcement learning from human feedback. The next generation of models, which are already in development at all the major labs, are being trained from the ground up on agentic trajectories: sequences of observations, thoughts, tool calls, and outcomes rather than simple text. These models will have a much more natural understanding of the agent loop and will require less scaffolding to operate reliably. This will shift the balance of the Agent = Model + Harness equation: the model will handle more of what the harness currently handles, and the harness will focus on the things that models are inherently bad at, like enforcing hard constraints and managing external state.
PROTOCOL MATURATION
MCP and A2A have established the foundation for a truly interoperable agentic AI ecosystem, but they are still young standards with rough edges. MCP in particular has significant security vulnerabilities in its current form, and the ecosystem of MCP servers is still largely unvetted. As these protocols mature -- through the Linux Foundation's governance process and through the accumulated experience of production deployments -- the harness engineer's job of integrating tools and coordinating agents will become progressively easier and safer.
SELF-IMPROVING HARNESSES
Hermes Agent's skill abstraction mechanism is an early example of a harness that improves itself over time. The next generation of self-improving harnesses will go further: using the agent's own execution traces to automatically identify weaknesses in the harness design, propose improvements, and in some cases implement those improvements autonomously. This creates a fascinating recursive loop: the harness helps the agent improve, and the agent helps the harness improve. The engineering discipline required to design and govern such systems safely is still being developed, and it will be one of the most important areas of harness engineering research in the coming years.
FORMALIZATION OF THE PROFESSION
Currently, there are no widely recognized certifications, no standard curricula, and no professional associations for harness engineers. This is beginning to change. The OWASP Top 10 for AI Agents represents the beginning of a security-focused professional standard. MLflow's evaluation framework represents the beginning of a quality assurance standard. The Linux Foundation's Agentic AI Foundation is developing governance standards for MCP and A2A. Within the next two to three years, we should expect to see the emergence of more comprehensive professional standards, training programs, and certification pathways for harness engineers.
EXPANSION INTO NON-CODING DOMAINS
Most of the harness engineering work to date has focused on software engineering agents, because software engineers were the first to adopt agentic AI tools and because the verification story for code (run the tests) is unusually clean. Agentic AI is rapidly expanding into other domains: scientific research, legal analysis, financial modeling, and healthcare administration. Each of these domains has its own verification challenges, its own governance requirements, and its own failure modes. Harness engineering for a medical information agent requires a very different approach from harness engineering for a coding agent: the verification criteria are different (clinical accuracy rather than test passage), the governance rules are different (HIPAA rather than SOC 2), and the escalation paths are different (physician review rather than developer review). The principles of harness engineering are universal, but their application is deeply domain-specific.
THE SHIFTING LOCUS OF INTELLIGENCE
Perhaps the most philosophically significant trend is the gradual shift in the locus of intelligence from the model to the harness. This might seem counterintuitive: surely the model is always going to be the smart part? But as harnesses become more sophisticated -- incorporating learned skills, self-improving mechanisms, and complex multi-agent orchestration -- the intelligence of the system becomes increasingly distributed between the model and the harness. A Hermes Agent deployment with a rich library of learned skills and a well-tuned orchestration strategy may outperform a more powerful model running in a poorly designed harness. This is the deepest implication of the Agent = Model + Harness equation: the two terms are not independent, and investing in the harness is often a more efficient path to better agent performance than investing in a more powerful model.
This observation leads to a final, somewhat provocative thought. The history of computing is, in many ways, a history of the operating system gradually absorbing capabilities that were once the exclusive province of specialized hardware or application code. Memory management, networking, security, scheduling: all of these started as things that individual programs had to handle themselves, and all of them eventually migrated into the OS because that is where they belong. The same process is happening with agentic AI. Capabilities that today's harness engineers implement from scratch -- memory management, tool orchestration, verification loops, governance controls -- will gradually be absorbed into the platform layer, just as operating system capabilities were absorbed from application code.
But here is the thing about operating systems: even as they absorbed more capabilities, the discipline of operating systems engineering did not become less important. It became more important, because the stakes were higher and the systems were more complex. The same will be true of Harness Engineering. As the platforms absorb more of the routine work, the harness engineer's job will shift toward the harder, higher-value problems: designing the governance architectures for systems that operate at planetary scale, building the verification frameworks for agents that work in high-stakes domains, and ensuring that the growing autonomy of agentic systems remains aligned with human values and organizational goals.
That is a discipline worth mastering. And it starts with understanding, deeply and precisely, what a harness is, why it matters, and how to build one well.
APPENDIX: REFERENCE SUMMARY OF KEY CONCEPTS, PLATFORMS, AND STANDARDS
This appendix collects the key concepts, platforms, protocols, and benchmarks discussed in this essay in a form that supports quick reference.
CORE CONCEPTS
The fundamental equation of Harness Engineering is Agent = Model + Harness, where the harness is the entire non-model runtime infrastructure that transforms a language model into a production-grade autonomous agent.
The seven layers of a harness are
the Instruction and Convention Layer,
the Tool and Integration Layer,
the Memory and Context Management Layer,
the Orchestration and Control Flow Layer, the Verification and Feedback Layer,
the Governance and Safety Layer,
and the Observability and Telemetry Layer.
Harness Engineering emerged as a named discipline in February 2026, succeeding Context Engineering (2025) and Prompt Engineering (2022-2024) as the dominant paradigm for building reliable agentic systems.
FRONTIER MODELS
Claude Fable 5.1 leads public intelligence rankings as of this writing.
GPT-6 Astra, which launched on September 3, 2026, holds a top-tier position with benchmark scores challenging Claude's lead.
Gemini 3.1 Pro leads on GPQA Diamond at 94.3 percent and on ARC-AGI-2 at 77.1 percent.
Claude Fable 5 leads on SWE-bench Verified at 95.0 percent.
Gemini 3.8 Flash provides high-value multimodal performance at significantly lower cost than frontier models.
Llama 4 Scout is among the fastest open-source models at approximately 2,600 tokens per second.
Grok 4.6 rounds out the top tier of general-purpose frontier models.
PLATFORMS AND HARNESSES
Claude Code (Anthropic) is a terminal-first coding agent harness featuring more than thirty lifecycle hooks, a sub-agent architecture, and deep MCP integration. Install it via `curl -fsSL https://claude.ai/install.sh | bash`.
OpenAI Codex is an Apache 2.0-licensed, Rust-based platform (Codex Core), operating as an open-source agent platform since August 2026; install via `npm install -g @openai/codex`.
OpenCode is an MIT-licensed, model-agnostic harness supporting more than seventy-five providers.
OpenClaw 2.0 is an open-source gateway platform with messaging platform integration and more than one hundred AgentSkills; install via `git clone https://github.com/openclaw/openclaw && pnpm install && pnpm build`.
Hermes Agent (Nous Research, MIT, launched February 25, 2026) is a self-improving agent with a closed learning loop and Atropos RL integration; install via `curl -fsSL https://hermes-agent.org/install.sh | bash`.
LangGraph is the enterprise production standard for directed graph-based state machine orchestration; install via `pip install langgraph langchain-anthropic`.
CrewAI provides role-based orchestration optimized for rapid prototyping; install via `pip install crewai`.
Microsoft Agent Framework 1.0 (GA April 2, 2026) is the unified successor to AutoGen and Semantic Kernel with YAML declarative agents; install via `pip install microsoft-agent-framework`.
Google ADK 2.0, operating within the Gemini Enterprise Agent Platform (rebranded from Vertex AI in April 2026), provides multi-language support and native A2A and MCP integration; install via `pip install google-adk`.
Pydantic AI 2.0 (June 23, 2026) introduces harness-first design centered on the capability primitive; install via `pip install pydantic-ai`.
PROTOCOLS
The Model Context Protocol (MCP), introduced by Anthropic in November 2024 and donated to the Linux Foundation's Agentic AI Foundation in December 2025, is the de facto standard for agent-to-tool connections, operating on JSON-RPC 2.0.
The Agent2Agent protocol (A2A), introduced by Google in April 2025, donated to the Linux Foundation in June 2025, and reaching v1.0.0 on March 12, 2026, is the de facto standard for agent-to-agent communication, supported by more than 150 organizations including Google, Microsoft, AWS, and IBM.
BENCHMARKS
SWE-bench Verified is the primary benchmark for software engineering agents, with Claude Fable 5 currently leading at 95.0 percent.
Terminal-Bench v4.0, the gold standard for command-line and system administration tasks, features sixty-six complex scenarios across diverse domains. tau-bench evaluates workflow and support tasks.
OSWorld evaluates GUI and computer operation tasks.
RE-Bench evaluates research engineering tasks.
SECURITY FRAMEWORK
The OWASP Top 10 for AI Agents is the industry's primary framework for categorizing agentic AI security risks, covering prompt injection, tool poisoning, memory poisoning, excessive agency, and goal hijacking. It should be used as a mandatory checklist for any production harness engineering project.
OBSERVABILITY AND EVALUATION
MLflow with genai.evaluate() is the industry standard for trace-based agent evaluation, providing built-in Agent GPA scoring, custom Python-based scorers, and production CI/CD integration. Install via `pip install mlflow mlflow-tracing`. OpenTelemetry-based instrumentation is the standard for agent tracing, with traces structured hierarchically: session, turn, agent loop, tool call, sub-operation.
