# The Model Is Rarely the Bottleneck: Anatomy of an Agent Harness

> The model is just one input into a running agent. The harness around it — prompt, tools, context, subagents — is the engineering discipline that decides whether an agent ships.

Author: Manik Soin (https://maniksoin.com)
Published: 2026-04-21
URL: https://maniksoin.com/blog/anatomy-of-an-agent-harness/
Tags: AI, Agents, Harness Engineering, Architecture

A lot of the debate about AI agents centres on the model: which one is smartest, which benchmark moved, which lab is ahead this quarter. That matters, but ship an agent to production and the lesson lands fast: the model is just *one input* into a running system. Everything else is the harness, the system prompt, the tools, the context policies, the hooks, the sandboxes, the subagents, the feedback and recovery loops. That's the quiet category error, and the harness is where agents live or die.

It's a view I've slowly come around to:

> A decent model with a great harness tends to beat a great model with a bad harness.

## The bottleneck isn't the model

Here's a figure that reframed things for me: by widely cited industry estimates, the great majority of AI agent projects (on the order of **80 to 90%**) never reach production. And anecdotally that gap hasn't obviously closed as the models got better. We went through several generational leaps in raw capability and the production rate still looks stubbornly low. If the model were the only bottleneck, you'd expect better models to have moved that number more than they seem to have.

To me that suggests the bottleneck often sits somewhere else: in the runtime orchestration layer wrapping the reasoning loop. That layer is the harness, and harness engineering is the discipline that seems to be emerging in 2026 to take it seriously.

The reframe that helped me: try not to treat the LLM as the agent. The LLM is a stateless function that maps `(prompt) -> (text)`. The agent is the loop, the state, the tools, and the policies that decide what goes into that function and what happens to what comes out. That's all harness.

```mermaid
flowchart TB
    subgraph HARNESS[The Harness, Runtime Orchestration]
        SP[System Prompt\nidentity + policies]
        TOOLS[Tool Registry\nname + schema + description]
        CTX[Context Policy\nwhat goes in, what gets pruned]
        SUB[Subagents\ndelegated sub-tasks]
        HOOKS[Hooks + Sandboxes\nguardrails + recovery]
    end
    SP --> LOOP
    TOOLS --> LOOP
    CTX --> LOOP
    SUB --> LOOP
    HOOKS --> LOOP
    LOOP[Reasoning Loop] --> MODEL[(LLM\none input among many)]
    MODEL --> LOOP
```

The model sits *inside* the harness, not above it. It's a component you can swap. The harness is the part I've found you spend most of your engineering effort on.

## The core loop: Thought-Action-Observation

Strip away the vocabulary and most agents I've worked on run the same loop. Call it TAO, call it ReAct, it tends to be the same cycle:

```mermaid
flowchart LR
    START[Goal] --> ASM[Assemble Prompt\nsystem + history + context]
    ASM --> CALL[Call LLM]
    CALL --> PARSE[Parse Output\nthought + tool calls]
    PARSE --> DONE{Done?}
    DONE -->|Yes| RESULT[Final Answer]
    DONE -->|No| EXEC[Execute Tool Calls]
    EXEC --> OBS[Observe Results]
    OBS --> ASM
```

1. **Assemble** the prompt: system instructions, conversation history, retrieved context, tool definitions.
2. **Call** the LLM.
3. **Parse** the output into a thought and zero or more tool calls.
4. **Execute** the tool calls against real systems.
5. **Feed** the observations back into the next prompt.
6. **Repeat** until the model decides the goal is met.

Simple to draw, hard to get right. Every iteration the harness makes decisions: what to include, what to drop, which tools to expose, what to do when a tool errors. Those decisions compound. A loop that runs 20 turns has made hundreds of small harness choices before it produces an answer.

## The four pillars

When I design a harness, I think about four things. Get these right and a lot of the rest follows.

**System prompt.** The agent's constitution: identity, operating principles, output contracts, escalation rules. This is where you encode the policies that don't belong in code because the model needs to reason about them.

**Tools.** The agent's hands. What it can actually *do* in the world.

**Context.** What the model sees on each turn. This is the pillar I've found hardest, and the one that has quietly broken the most agents I've built.

**Subagents.** Delegation. When a task has a self-contained sub-goal, you spin up a fresh agent with its own clean context and a narrow toolset, let it finish, and fold the result back. It's how you keep the main loop's context from drowning.

Hooks, sandboxes, and recovery paths matter too, but I tend to treat them as the guardrail layer wrapping these four, not a fifth pillar. Get the four right and they're what you're guarding.

## Tool design is prompt design

Here's something that's easy to miss: every tool's name, description, and JSON schema gets **stamped into the prompt on every single request**. Tools aren't free function calls sitting off to the side; they're tokens the model reads every turn, and they shape how it reasons.

This is why **ten focused tools beat fifty overlapping ones.** With ten clean tools, the model can hold the entire menu in its head and pick correctly. With fifty, you start to see tool confusion: `search_documents` versus `find_files` versus `query_index`, and the model coin-flips. The menu is part of the prompt, so a bloated menu is a bloated prompt.

I've found it helps to treat tool design as prompt engineering. Name tools the way you'd name them for a new engineer. Write descriptions that say when *not* to use them. Make the schemas tight so invalid calls are hard to represent.

## The two failure modes that actually kill agents

After a fair number of production incidents, the failure modes I keep running into sort themselves into a rough ranking.

**Context rot** is the one I see most. As the loop runs, context accumulates: old observations, stale tool outputs, dead ends the agent explored and abandoned, half-relevant retrieved chunks. None of it gets cleaned up by default, and it all tends to degrade the next generation. The model starts attending to noise. Output quality drops turn over turn, even though nothing "errored."

The mental model that has helped me most: **treat context rot as a sensor problem.** Your agent perceives the world through its context window. If the window fills with stale and noisy signal, the agent is effectively perceiving a degraded version of reality. You wouldn't run a robot off a fogged-up camera; don't run an agent off a fogged-up context.

**State loss** is the second one I hit often. The agent forgets what it already did, redoes work, contradicts an earlier decision, or loses the plot on a long task. The context window is finite and the loop is long, so eventually something important scrolls out of view.

Both failure modes are quiet, nothing errors, so I've found it helps to name them precisely:

| Failure mode | Symptom | Cause | Fix |
| --- | --- | --- | --- |
| Context rot | Output quality drops turn over turn, no errors thrown | Stale observations, dead ends, and half-relevant chunks accumulate and never get pruned | Progressive disclosure: surface only the minimal slice the agent needs right now |
| State loss | Agent redoes work, contradicts itself, loses the plot on long tasks | Finite window plus long loop, so important state scrolls out of view | Externalize memory to the filesystem (AGENTS.md, todo.md) and re-read on demand |

## Mitigations: progressive disclosure and externalized memory

Two techniques have done most of the heavy lifting for me against both failure modes.

**Progressive disclosure.** Don't expose everything up front. For each sub-task, surface the *minimum* set of tools and telemetry the agent needs right now. Drilling into logs? Reveal the log tools then, not before. This keeps the per-turn prompt lean and directly fights context rot: less surface area, less noise, fewer ways to get confused.

**Externalize memory to the filesystem.** Instead of trying to cram all state into the context window, push it out to standardized artifacts the agent reads and writes: an `AGENTS.md` describing the environment and conventions, a `todo.md` tracking the plan and what's done. The context window becomes working memory, scoped to the current turn; the filesystem becomes long-term memory, durable across the whole run. The agent re-reads its `todo.md` instead of trying to remember a 40-step plan from 30 turns ago. That's the state-loss fix.

```mermaid
flowchart TB
    LOOP[Reasoning Loop] -->|reads minimal slice| CTX[Context Window\nworking memory]
    CTX -->|prune stale signal| CTX
    LOOP -->|write plan + progress| FS[(Filesystem\nAGENTS.md / todo.md)]
    FS -->|re-read on demand| LOOP
    LOOP -->|progressive disclosure| TOOLS[Per-task Tool Subset]
```

## The MCP angle

One reason harness engineering seems to be crystallizing as a discipline now is the **Model Context Protocol (MCP)**. MCP standardizes how agents talk to tools; it draws a clean line between *tool implementation* and *agent logic*. Your agent doesn't need to know how a tool is built; it speaks a common protocol and the tool answers. That separation is what lets you treat the tool registry as a first-class, swappable layer of the harness instead of bespoke glue welded into the loop.

## Where this connects to my own work

I wrote earlier about building [Ordinance](/blog/building-rag-for-hk-regulations), my RAG system for Hong Kong building regulations. That post was about retrieval: hybrid search, RRF fusion, citation verification. But stepping back, the questions that turned out to matter most for me were harness questions: which retrieval tools to expose per query type, how to keep multi-hop regulatory reasoning from rotting its own context, when to spin up a subagent to chase a cross-reference, how to externalize the audit trail as durable state. The retrieval was, for me, the more tractable half. The harness was the next half, which is exactly what I built next, written up in [From RAG to Agent](/blog/from-rag-to-agent-harnessing-ordinance).

That's the through-line I keep coming back to. The model keeps getting better and that's great, but it was rarely the binding constraint. The constraint is everything around it, and that everything has a name now. Build the harness like it's the product, because to your users, it largely is.