Why We Cap Context Instead of Filling It
We once handed a customer system 40 passages because they fit in the window — and got worse answers than with twelve. A paper by Liu et al. explained why, and we have treated position as a quality parameter ever since.
The Problem, Before We Had a Name for It
The system was knowledge access over a large document collection. Retrieval measured well: the correct passage was among those retrieved in more than nine cases out of ten. The answers were still unreliable. Sometimes the information sat verbatim in the context we had handed over, and the model answered past it.
Our first guess was the usual suspect: not enough context. So we handed over more. It got worse. That was the point at which it became clear we were optimising the wrong quantity.
What We Had Missed
We had treated retrieval as a question of coverage: is the right passage in the set, yes or no? That answers only half the chain. The second half is whether the passage is actually used once it is there. And that depends on where it sits.
This is what "Lost in the Middle" by Liu et al. measures. The authors move the relevant passage systematically through the context and hold everything else constant. The result: performance is highest at the beginning and the end of the input, and degrades significantly when the model has to reach into the middle. The addition that mattered most to us is that this shows up even in models built explicitly for long contexts.
fig. — hit rate by positiongold: the passage we need · cyan: what we do about it
What We Changed
Three things, in this order. First we capped how many passages we hand over. Not "as many as fit" but as few as needed — twelve instead of forty in the system above. That alone produced the largest jump, without touching the search at all.
Second we put cross-encoder reranking in front of the hand-off. The point of it is not the better selection but the better ordering: whatever gets read most reliably belongs at the edges. Third we stopped measuring retrieval quality as recall alone.
How We Test It Now
Our test set carries a variation many teams do not have: same question, same documents, only the answer's position moves. Whatever difference remains is a position effect and nothing else. It is cheap to build and exposes a failure class that looks like randomness in normal operation.
We work to a rule of thumb: if answer quality differs by more than a few percentage points between position 1 and position 20, we are handing over too much. Capping is then the cheaper correction than a bigger model.
Where We Decide Differently on Purpose
The effect is not a reason to keep context small as a rule. For tasks that require summarising a whole document, more context is right — there is no single spot that can get lost. So we separate two cases: answers drawn from one passage get capped, aggregation across many does not.
Nor does it follow that long contexts replace retrieval. Rather the opposite: if position co-determines the outcome, selection and ordering matter more — which is exactly what a retrieval stage provides.
What We Do Not Take From the Paper
| Common reading | Our view |
|---|---|
| Long contexts are useless | No. What is measured is a position effect, not a total failure |
| The next model generation will fix it | Open. The paper shows the effect in long-context models too; we do not plan on it disappearing |
| The cause is understood | No. The behaviour is measured, the explanation is not settled. We build against the behaviour, not against a theory |
The Cost, Honestly Counted
Reranking costs one additional model call per query, so tens of milliseconds of latency and some compute. Capping the passage count saves input tokens, often more than the reranking costs. Across the systems we operate, the change was not more expensive overall — only more reliable.
What We Take Away
A context window is a capacity figure, not a quality promise. What fits is not the same as what gets used. Since we started treating position as its own parameter — cap, order, test — we argue far less often about switching models when answers go wrong.