Engineering notes
Every article starts with a practical question, explains the necessary terminology, and links to primary sources. We separate external evidence, project experience, and conclusions — and state what cannot be inferred from them.
Four dated releases define this 2024 retrospective: OpenAI Structured Outputs in August, o1-preview in September, Anthropic computer use in October, and the Model Context Protocol in November. We summarize the verified numbers, the limits each capability still has, the lessons production teams drew from them, and our predictions for 2025.
OpenAI's o1 made test-time reasoning the frontier capability of late 2024 — closed, expensive, and API-only. Yet DeepSeek's R1-Lite-Preview and Alibaba's QwQ-32B-Preview signal that open alternatives are weeks away, not years. We examine what reinforcement-learned reasoning actually does, what open weights would do to its cost structure, and why that matters for European data sovereignty.
A customer asked us to replace a half-time position with an agent. We listed the eleven tasks that position actually performs and priced each one: four were worth automating, three were already someone else's job in disguise, and four depend on judgement we would not delegate. Why benchmark completion rates on real workplace tasks are the number to quote, and what the leftover coordination costs.
Anthropic open-sourced the Model Context Protocol: a JSON-RPC standard that connects AI applications to data sources through servers, clients, tools, resources, and prompts. We explain the architecture, the stdio transport, what the protocol deliberately leaves unsolved, and why one integration surface could replace N-by-M custom connectors.
LazyGraphRAG from Microsoft Research shifts all LLM summarization from indexing to query time. Indexing costs drop to 0.1% of full GraphRAG — identical to vector RAG — while answer quality matches or exceeds GraphRAG global search at a fraction of the query cost. We examine the mechanism, the benchmark numbers, and the trade-offs for production systems.
We put a person in the loop and called it solved. Approval rates by day of month showed the check working for three weeks and collapsing in the fourth, when volume and pressure peak. Haduong et al. study what pressure does to AI-assisted decisions. What we changed in the queue rather than in the people, and why we now treat a human check as a component with a capacity limit.
Three weeks of work cut median response time by a third and the feedback did not change. Users do not experience total duration, they experience the silence before anything appears, and our retrieval and reranking ran entirely inside that silence. What we reordered, why a progress message beats a spinner, and the case where streaming is the wrong answer.
Despite measurable 2024 progress in AutoGen, CrewAI, and LangGraph, multi-agent systems still pay a steep tax: coordination overhead, compounding errors, and fragmented shared context. We survey the framework landscape as of November 2024, quantify where the overhead comes from, and argue that a single agent with good tools remains the right default for most production tasks.
A customer showed us an agent operating their ERP through its interface and asked how fast we could ship it. A screen is a rendering of a system, not the system, and automating against it means depending on a layout nobody promised to keep. The three questions we ask first, the one project where the answer was yes, and what we insisted on before it ran unattended.
Anthropic released computer use in public beta on October 22, 2024: Claude 3.5 Sonnet operates desktops via screenshots, mouse, and keyboard. We explain the agent loop, the OSWorld results (14.9 percent versus roughly 72 percent for humans), known failure modes, safety gating, and which workflows this technique should and should not automate today.
Renaming one field broke a step that had passed eighty cases for months, with nothing about the task changed. The cases were real and they were also all phrased the same way, so the set had been measuring a step's fit to our phrasing rather than to the task. What we generate now to keep it honest, and why this is the one place model-generated data earns its place.
Phi-3 and Gemma 2 show that models with 3 to 9 billion parameters can carry a large share of production traffic at a fraction of frontier cost. We identify which tasks small models cover, what on-premises and edge deployment requires, how routing directs easy traffic to cheap models, and how to measure quality per euro.