Engineering notes
Every article starts with a practical question, explains the necessary terminology, and links to primary sources. We separate external evidence, project experience, and conclusions — and state what cannot be inferred from them.
A customer asked what their share of our model spend was, and we could not answer: one invoice, no attribution. The gateway we built took a week and repaid it four other ways, including the one that mattered most, which is that a model swap became configuration instead of a change in eleven places. What it records, what it refuses, and the one thing it must never become.
A shared answer cache cut costs by about a fifth and worked exactly as designed. It also meant a fast response revealed that someone else had recently asked a similar question, which for a platform serving competing companies is a disclosure nobody agreed to. Where we cache now, why per-tenant caching keeps most of the saving, and the timing channel that survives even that.
Half our users ask in German against documentation written in English, and for a long time that failure looked like poor coverage rather than a language problem. We measured four ways to bridge it: translating queries, translating documents, indexing both, and a multilingual embedding model. The obvious one came third, and the winner depends on something most comparisons leave out.
Released September 12, 2024, OpenAI's o1-preview and o1-mini trade latency and cost for accuracy by spending compute on hidden reasoning tokens at inference time. We examine where trained deliberation helps — math, code, planning — what it costs, why the chain of thought stays hidden, and what test-time compute means for engineering.
A context window is working memory, not durable memory. This article separates the two concepts and reviews the three memory mechanisms in production use for LLM agents in 2024 — conversation summaries, vector memories and structured state — with their failure modes, why deliberate forgetting improves correctness, and which boundaries the GDPR sets for persistent agent memory.
We do not train models, so we had filed training-time risk under not our problem. An inventory found eleven model and embedding artefacts pulled from public registries into systems we run, with provenance we could name for four. What we pin, what we mirror, why a hash is the cheapest control available, and the honest limit: we cannot inspect weights, so our defences are all about blast radius.
A tool call passed every check we had: valid JSON, right field names, matching types, and an order number that had never existed. Shape validation and reference validation are different questions, and only one of them was being asked. What we check now before a call leaves the process, why we stopped letting the model retry blindly, and the argument that fails in a way no schema can catch.
One step in a nine-step pipeline was failing, and the obvious fix was a bigger model for everything. Snell et al. show that spending more compute at inference can beat spending it on model size. We spent it on one step: three candidate plans, a deterministic check on each, first valid one wins. It cost about three percent of what upgrading the whole system would have.
Anthropic's prompt caching beta changes the cost structure of long prompts: cache writes cost 25 percent more than base input tokens, cache reads 90 percent less. We walk through the pricing, the break-even arithmetic, the limits of the five-minute TTL, and what the numbers mean for RAG pipelines and agent loops.
Every few months someone asks whether long context windows make retrieval unnecessary. It is an arithmetic question. For one of our corpora the whole thing fits and we do put it in the window; for another it would have cost forty times more per question for a quality difference we could not measure. The three properties that decide it, and the hybrid we run where the corpus sits in between.
Introduced in June 2024, ColPali replaces the OCR parsing pipeline with direct embedding of document page images. We explain the late-interaction mechanism inherited from ColBERT, the ViDoRe benchmark results — 81.3 versus 66.1 nDCG@5 — and the storage and scaling trade-offs, and assess where visual retrieval fits in production RAG systems as of July 2024.
A supplier pasted a note for a colleague into the footer of a delivery note, our extraction step read it as an instruction, and followed it. Nothing malicious had to happen for us to discover we had no boundary between content and command. What we changed, why filtering for injection patterns is the weakest of the four measures, and the one that actually holds: content never reaches a step that can act.