Context engineering: why your AI agent fails long before the window is full
Every model vendor now advertises a context window measured in hundreds of thousands of tokens, and every team that has actually shipped an agent has learned the same thing the hard way: the model gets worse long before that window fills. This is the full tutorial on context engineering — what is really inside the payload you send, the four distinct ways long context fails, the four levers that fix them, and how to tell whether your context pipeline is working before your users do.
The 30-second version
- A context window is a capacity limit, not a performance guarantee. Chroma tested 18 frontier models and every one degraded as input grew — well below the advertised limit.
- Your context is not one blob. It is six slots with six different owners, and most teams only control two of them.
- Long context fails in four distinct ways — poisoning, distraction, confusion and clash. They look alike from the outside and need completely different fixes.
- There are only four levers: write, select, compress, isolate. Each one answers a specific failure.
- More context is not better context. The job is relevance density — the share of what you send that the model actually needed.
1. The window is not the budget
Here is the conversation that happens in almost every team building an agent. Something breaks. Someone says the model is not smart enough. Someone else says the prompt needs work. A third person points out that the context window is a million tokens and they are only using sixty thousand, so surely capacity is not the problem.
That last argument is the one worth killing first, because it sounds like evidence and it is not. The window tells you what fits. It says nothing about what the model can still reason over.
Context length is a capacity limit, not a performance guarantee. Performance starts falling from the first tokens you add, gradually and continuously, and it does it while the model keeps producing confident, fluent output. Nothing errors. The answers just quietly get worse.
The research here is unusually clear. Chroma's Context Rot study evaluated 18 frontier models — across the GPT, Claude, Gemini and Qwen families — and found that every single one performed worse as input length increased. Not at the limit. Throughout. The degradation was also non-uniform: it depended on where the relevant information sat, whether semantically similar distractors were present, and how the surrounding material was structured.
Two findings from that work are worth holding onto, because they overturn the intuition most people are running on:
- A single distractor hurts. One plausible-but-wrong passage measurably reduced performance against a clean baseline. Four compounded it. You do not need a flood of noise to degrade an answer — you need one convincing near-miss.
- Models did better on shuffled haystacks than logically coherent ones. Which is deeply counterintuitive, and a strong hint that "just give it the whole document, it will find what it needs" is not the safe default it feels like.
Older work points the same direction with blunter numbers. A 2023 Stanford study found that with roughly 20 retrieved documents — about 4,000 tokens, nothing by modern standards — accuracy fell from around 70–75% to 55–60%. Position mattered independently: the same fact placed first yielded about 75% accuracy, and the same fact placed tenth yielded about 55%.
Read that again, because it is the whole argument for this discipline. The identical information, in the identical payload, produced a twenty-point accuracy swing based on nothing but where it sat. If that is true — and it has been reproduced repeatedly since — then assembling the payload is not plumbing. It is the work.
A crashed agent tells you it crashed. A context-degraded agent returns a well-formed, plausible, subtly wrong answer, at the same latency as a good one, with no error in your logs. Teams routinely run in this state for months and attribute it to "the model being flaky."
This is also why the framing shifted. If you have read our tutorial on loop engineering, you will recognise the layer diagram: prompt sits inside context, context sits inside the loop, the loop sits inside the harness. This post is the deep dive on the second layer — the one that decides what the model sees on any given turn.
2. What is actually in your context
Ask most teams what is in their context and you get an answer about the prompt. The prompt is usually the smallest part. On a mature agent turn, the payload has six distinct slots, and the striking thing is how few of them the prompt author controls.
| Slot | Who fills it | How it goes wrong |
|---|---|---|
| System instructions | You, once | Accretes rules over months; contradictory clauses nobody removed |
| Tool definitions | Your integrations | Every tool you add is charged on every turn, whether relevant or not |
| Conversation history | The loop, automatically | Grows unbounded; failed attempts persist and get re-read forever |
| Retrieved documents | Your retriever | Top-k returns k things whether or not k things are relevant |
| Tool output | Whatever API you called | A 40-line JSON response lands whole; nobody budgeted for it |
| Scratchpad / state | You, if you built one | Usually missing, so state hides in the transcript instead |
The useful exercise: take one real production turn, dump the exact payload, and attribute every token to one of these six rows. Almost everyone is surprised by which row wins.
That attribution exercise is the single highest-value hour in this whole discipline. We have run it on client systems and found agents spending 60% of every payload on tool definitions for tools that turn could not possibly need, and others where a single verbose API response was consuming more of the window than the entire task description.
3. Four ways long context fails
"The context got too long" is not a diagnosis. It is four different problems wearing the same coat, and they have different fixes — so naming yours correctly is most of the work. This taxonomy comes from Drew Breunig's write-up on how contexts fail, which has become the shared vocabulary for this.
Poisoning
Something false enters the context and is then treated as established fact by every subsequent turn. The agent is no longer reasoning from reality — it is reasoning from a corrupted premise it wrote itself.
Distraction
Accumulated history grows heavy enough to crowd out fresh reasoning. The agent starts pattern-matching its own past behaviour instead of responding to the current state of the world.
Confusion
Irrelevant material — most often tool definitions the turn will never use — degrades the decision the model makes. Not because it is wrong, but because it is there.
Clash
Two pieces of context contradict each other. The model cannot flag ambiguity the way a person would, so it silently picks one and proceeds as if there were never a conflict.
Poisoning: the error that becomes a fact
The clearest documented example came out of the Gemini 2.5 Pokémon experiments. The agent maintained a running "goals" and "summary" section in its context — a sensible design. When a hallucination made it into that section, the effect was not a single bad turn. The model became fixated, pursuing goals that were impossible or irrelevant, because the corrupted summary was now the most authoritative-looking thing in its own context.
Poisoning is the most dangerous of the four precisely because agents are designed to trust their own accumulated state. A retrieval error affects one answer. A poisoned summary affects every answer after it, and it compounds — the agent writes new reasoning on top of the false premise, which makes the false premise look increasingly well-supported.
The agent keeps referring to something you cannot find in any source document. If you ask it where that came from, it will confidently cite its own earlier turn.
Distraction: repeating instead of reasoning
As history accumulates, models begin favouring repetition of past actions over novel reasoning. In the Gemini 2.5 Pro observations, this kicked in as context grew significantly beyond 100k tokens — the agent showed a marked tendency to repeat actions it had already taken rather than synthesise a new approach. Databricks found correctness starting to decline around 32k tokens for Llama 3.1 405b, which is a useful reminder that the threshold scales with the model, and is often much lower than you would guess.
The important nuance: this is not the model forgetting. It is the model over-weighting. The history is right there, in full, and that is exactly the problem — a large body of past actions is a very strong prior, strong enough to outweigh what the current state is telling it.
Confusion: paying attention to things that do not matter
This one has the most actionable evidence attached, and it is about tools. The Berkeley Function-Calling Leaderboard found that every model performs worse when given more than one tool. Not "more than thirty" — more than one. Degradation grows non-linearly as tool count rises, and hits smaller models hardest.
A GeoEngine benchmark run made it concrete: a quantised Llama 3.1 8b failed the task with all 46 tools present, and succeeded with only 19 provided. Same model, same task, same prompt. The only variable was how many irrelevant tool definitions were sitting in the window.
If your agent has a tool registry that grows as your product grows, and every tool is passed on every turn, you have built a system that gets measurably worse each time you ship a feature. Tool selection is not an optimisation — it is a correctness requirement.
Clash: two truths, no flag
The Microsoft and Salesforce study on multi-turn conversation is the sharpest data point here. When a single complete prompt was sharded across sequential turns — the way real users actually talk — performance dropped by an average of 39%. OpenAI's o3 fell from 98.1 to 64.1 on the same underlying task.
The mechanism is what matters: the model's early, under-informed attempts stayed in the context, and those wrong early answers then contaminated the final reasoning. The researchers' conclusion was blunt — when LLMs take a wrong turn in a conversation, they get lost and do not recover.
This is the failure mode most likely to be in your product right now, because it needs no long documents and no big tool registry. It only needs a user who did not state the full requirement in message one. Which is every user.
| Failure | Symptom you will actually observe | The lever that fixes it |
|---|---|---|
| Poisoning | Confidently cites a fact that exists in no source | Write — external, validated state |
| Distraction | Retries the same failing approach; ignores new evidence | Compress — summarise and prune history |
| Confusion | Picks the wrong tool; quality drops as you add features | Select — just-in-time tool and doc loading |
| Clash | Degrades over a conversation; early wrong guess sticks | Isolate — fresh context, restated requirement |
Match the lever to the failure. Applying compression to a poisoning problem just produces a shorter poisoned context.
4. Write, select, compress, isolate
Against those four failures there are exactly four things you can do to a context. Everything else is a variation. This framing — write, select, compress, isolate — has become the standard toolkit, and its value is that it turns a vague instruction ("manage your context better") into four concrete places to put engineering effort.
- Write — move state out of the window Anything the agent needs to remember does not have to live in the transcript. Write it to a file, a scratchpad, a database row, a structured plan document. The transcript then carries a pointer instead of the payload. This is the answer to poisoning, because external state can be validated, versioned and corrected — a transcript cannot.
- Select — pull in only what this turn needs Retrieve documents per turn rather than front-loading them. Expose the three tools this step could plausibly use rather than all forty-six. Load the schema for the table being queried, not the whole catalogue. Selection is where the Berkeley tool-count finding gets paid off.
- Compress — make history cost less than it did Summarise completed sub-tasks into their outcome. Prune tool outputs to the fields that mattered. Replace a resolved twelve-turn debugging exchange with one line recording what was learned. Done well, the agent keeps the lesson and drops the transcript.
- Isolate — keep unrelated material apart Give a sub-task its own clean context, hand back only the result. This is what subagents are actually for — not parallelism, but the guarantee that the research agent's forty dead ends never enter the writing agent's window. Isolation is the structural answer to clash.
If you only do one of these: select. It has the best evidence behind it, it usually requires no architectural change, and cutting an unused tool registry from every turn is often a same-day fix with a measurable quality jump.
Compression has a failure mode of its own
Summarisation is the most popular lever and the most dangerous one, because a summariser is a model, and a model can hallucinate. A summary that invents a detail is a poisoning event that you scheduled. Worse, it is poisoning with the transcript deleted, so nothing remains to contradict it.
The mitigations are unglamorous and they work: summarise into a fixed schema rather than free prose, keep the raw transcript retrievable rather than discarded, never let a summary be the only record of a decision, and treat the summariser as a component you evaluate like any other — see our guide to AI evals for how to build that test set.
5. Retrieval is context engineering
If you have read what is RAG, you already know the mechanics of retrieval. What is worth adding here is a change of framing: RAG is not an alternative to context engineering — it is one implementation of the "select" lever, filling one of the six slots.
That reframing has practical consequences, and they mostly amount to abandoning defaults that were never examined:
- Top-k is a bad default. Fixed k returns k chunks whether or not k chunks are relevant. Use a relevance threshold, and be willing to return two chunks — or zero.
- Returning nothing is a valid, valuable outcome. An empty retrieval that lets the agent say "I don't have that" beats five weak chunks that let it construct something plausible.
- Rerank, then truncate. The Stanford position finding means the ordering of what you pass is a correctness decision, not a presentation detail. Put the strongest evidence first.
- Near-misses are worse than noise. Chroma found semantically similar distractors did disproportionate damage — precisely the chunks a naive vector search loves to return.
- Structure beats volume. Where relationships matter, a small traversal returns less and answers better than a large similarity sweep. That is the argument in our graph engineering tutorial.
The instinct being corrected throughout is the same one: that the retriever's job is to be generous. It is not. Its job is to be right, and to be honest when it has nothing.
6. What survives a turn
Single-turn context engineering is mostly retrieval. Multi-turn is where the genuinely hard decisions live, because now you must decide — explicitly, every turn — what carries forward and what does not.
Most systems never make that decision. They append. Every message, every tool call, every result, forever, until something truncates from the front and silently drops the requirement the user stated in message one. That is not a memory policy; it is the absence of one.
| Carry forward every turn | Summarise after it resolves | Drop or externalise |
|---|---|---|
| The stated goal, verbatim | Completed sub-tasks → outcome only | Raw tool payloads, post-extraction |
| Hard constraints and preferences | Resolved debugging exchanges → the lesson | Superseded plans and drafts |
| Decisions already made | Long documents → the relevant passage | Retrieved docs the turn did not use |
| Open questions | Failed approaches → one line each | Definitions for tools not in scope |
Column one is the part teams get wrong most often: the original requirement should be re-stated near the end of the payload, not left to decay at the top of a growing transcript.
That last point deserves emphasis. Given the position effect, the goal drifting toward the beginning of an ever-longer context is a slow-motion failure with a trivial fix: restate it, close to the instruction, on every turn. It costs a few dozen tokens and it removes an entire class of drift.
Pick a threshold — a token count, a turn count, a completed milestone. When you cross it, compact deliberately: goal, constraints, decisions, open questions, current state. Five fields. Then continue. A scheduled compaction you designed always beats a truncation the API performed for you.
7. How to measure context quality
Context engineering fails as a discipline the moment it becomes taste. These four measures make it an engineering activity instead, and none of them require anything more exotic than logging your payloads.
Context precision
Of the tokens you sent, what share was relevant to the answer? Low precision is the direct cause of confusion and distraction.
Context recall
How often was the fact needed to answer actually present? This separates a retrieval problem from a reasoning problem — and they get confused constantly.
Position of the needed fact
Where in the payload did it land? Given the accuracy swing between first and tenth place, this is worth logging on every turn.
The fourth is the simplest and the one we would add to any agent first: plot tokens per turn across a run. A healthy loop's context grows and then flattens as compaction takes hold. A line that climbs steadily while the task makes no progress is distraction forming in real time, and you can see it long before the output gets bad enough for a user to complain.
Precision and recall pull against each other on purpose — you can always raise recall by sending more, at the cost of precision. Tracking both stops you from optimising one into the ground. And track them per turn, not per run: averaging across a run hides the single turn where the context broke, which is exactly the turn you need to look at.
8. The production context pipeline
Assembled, the levers become a pipeline that runs before every model call. The shape matters more than the specific implementation: context assembly is a deliberate stage in your system, with its own code and its own tests, rather than something that happens by accident in whatever order your appends ran.
- Budget Decide the ceiling per slot before the turn runs — system, tools, history, retrieved, output, scratchpad. A budget you set is a constraint; a budget you discover at truncation time is an incident.
- Select Choose the tools this step could plausibly need. Retrieve against a relevance threshold, not a fixed k. Prefer returning nothing over returning weak matches.
- Validate Check retrieved content against its source before it enters the window. This is the cheapest anti-poisoning measure available, and almost nobody does it.
- Compact If history has crossed the threshold, summarise into the fixed five-field schema — goal, constraints, decisions, open questions, state — and keep the raw transcript retrievable elsewhere.
- Order Rerank by relevance. Strongest evidence first, goal restated near the instruction at the end. Position is a correctness decision.
- Log Record the assembled payload with per-slot token counts. Without this, every later debugging session is guesswork about what the model actually saw.
Take a turn your agent got wrong. Replay it with the payload you would now assemble. If the answer improves without touching the model or the prompt, your context was the bug — and you now have a regression test.
9. Fixing your worst turn this week
You do not need a rebuild to get most of the value here. This is a two-hour exercise that reliably finds something:
- Dump one real payload Take a production turn that went wrong. Log the exact bytes sent to the model — not your template, the resolved payload.
- Attribute every token Assign each one to a slot from section 2. The largest slot is rarely the one you expected, and it is usually tool definitions or unpruned tool output.
- Name the failure Poisoning, distraction, confusion or clash. Pick one. If you cannot pick, look for the near-miss distractor — it is normally hiding there.
- Apply the matching lever One lever, one change. Re-run the same turn. Measure the difference against the wrong answer you started with.
- Keep it as a test That payload plus the correct answer is now a regression case. Ten of them is a context eval suite, and it will catch the next regression before your users do.
10. Glossary
- Context window
- The maximum tokens a model accepts in one call. A capacity limit, not a promise about quality at that capacity.
- Context rot
- The gradual, continuous degradation in output quality as input length grows, well before any limit is reached.
- Relevance density
- The share of the payload that was actually needed. The number context engineering exists to raise.
- Compaction
- Deliberately summarising accumulated history into a fixed schema at a chosen checkpoint, rather than letting truncation happen to you.
- Distractor
- Content that is semantically close to the answer but wrong. More damaging than unrelated noise.
- Just-in-time loading
- Fetching tools, schemas or documents at the turn that needs them instead of front-loading everything.
- Isolation
- Running a sub-task in its own context and returning only the result, so intermediate work never enters the parent window.
Frequently asked questions
What is context engineering, in one paragraph?
Doesn't a bigger context window solve this?
How is this different from RAG?
My agent works for a few turns then degrades. Which failure is that?
How many tools is too many?
Is summarising history safe?
Where this leaves you
The models are not the constraint any more. That is the genuine shift behind all the noise, and it is why this became the loudest topic in AI engineering rather than another framework debate. When the model is good and the system is still unreliable, the gap is almost always in what the model was shown.
The good news is that context is the most tractable layer in the stack. You cannot retrain the model, and rewriting the harness is a project. But dumping one payload, attributing its tokens and cutting an unused tool registry is an afternoon — and it is frequently worth more than a model upgrade.
If you are running an agent in production that has quietly got less reliable as you added features, that pattern has a name and a fix. Auditing context pipelines and rebuilding them into something measurable is exactly the kind of work we do.
Talk to us about your AI project
Ashish M.
Two decades of building and shipping software — from early e-commerce storefronts to multi-tenant SaaS, AI/RAG systems and marketplace automation. Hands-on with architecture, delivery and the long-term partnerships that keep clients with us for years.
Connect on LinkedIn →