Engineering notes
Every article starts with a practical question, explains the necessary terminology, and links to primary sources. We separate external evidence, project experience, and conclusions — and state what cannot be inferred from them.
Released April 18, 2024, Meta's Llama 3 narrows the gap between open weights and closed APIs. We examine the 8B and 70B benchmarks, what 4-bit quantization does to hardware requirements, the utilization math of self-hosting versus API pricing, and the privacy-bound workloads where an open model is the only viable option.
The scan was clean, character recognition was accurate, and the extracted data was wrong: a three-column table read left to right produces correct words in an order that means something else. Where we now send the image instead of the text, why we kept OCR for most documents anyway, and the one check that makes either path safe to run unattended.
Released 11 March 2024, Cohere's Command R treats grounded generation as a trained capability rather than a prompt trick: span-level citations, native tool use, 128k context, and open research weights. We examine what a RAG-tuned model changes for production systems, what it does not solve, and where the category is heading.
The change was one line of configuration. The consequence was half an index answering in the old vector space and half in the new one for most of a working day, with a symptom that looked like a quality problem. Why vectors from two models cannot be compared at all, the dual-index procedure we use now, and why a better benchmark score is not on its own a reason to pay for a re-index.
Agentic workflows replace single-pass prompting with iterative loops: reflection, tool use, planning, and multi-agent collaboration. We compare the measured gains behind each pattern — from Reflexion's HumanEval results to Devin's SWE-bench score — state what the techniques do not deliver, and argue for starting with the simplest loop that an evaluation can verify.
A customer asked why we were not using the model at the top of a ranking. We ran the swap on our own set and the higher-ranked model was worse at the work we do: it wrote better prose and followed the schema less reliably, which is exactly the trade a preference ranking rewards. What a leaderboard measures, what it structurally cannot, and how eighty of our own cases decide instead.
Anthropic's Claude 3 family replaces the single-model default with three tiers — Opus, Sonnet, and Haiku: one API, three price points, vision input, and a 200K context window across all tiers. We outline the verified launch numbers, the trade-offs, and a practical routing pattern that assigns each workload the cheapest model passing its evaluation.
Gemini 1.5 Pro accepts up to one million tokens of context. Does that make retrieval-augmented generation obsolete? We run the numbers: per-query cost, latency, multi-needle recall, freshness, and access control. The conclusion from February 2024: long context changes how retrieval pipelines are built — it does not replace them.
Nobody decided to send twenty thousand tokens of context. It accumulated in small defensible increments, and quality fell without any single change being the cause. Fitting inside the window is not the same as being usable, the material that hurt us was relevant rather than junk, and a budget only works if adding something means removing something.
Released on 25 January 2024, OpenAI's text-embedding-3 models support shortening vectors via a dimensions parameter. We explain the technique behind it — Matryoshka Representation Learning — verify the benchmark numbers, calculate storage savings at scale, and show why migrating from ada-002 still means re-embedding the entire corpus.
A classification step disagreed with itself on six percent of inputs. Five runs and a majority vote removed it. The same trick on a summarisation step bought nothing at five times the cost. What separates the two is whether the outputs can be compared at all, and we have a two-line rule for it now. Also the harder case: a wrong answer that repeats is not a right one.
A three hundred page manual retrieved perfectly for every torque figure and failed on what the machine was for. The answer to a global question is not in any one passage, so no amount of better ranking finds it. What we added, what a summary layer costs to keep honest when the source changes, and why we only build one where the questions actually justify it.