Engineering notes
Every article starts with a practical question, explains the necessary terminology, and links to primary sources. We separate external evidence, project experience, and conclusions — and state what cannot be inferred from them.
In 2024, vision LLMs turn invoices and contracts into schema-validated JSON. We assess what GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro deliver on documents, which accuracy ranges are realistic in production, why the validator is written before the prompt, and where human review remains structurally necessary.
Microsoft published GraphRAG: an indexing pipeline that uses an LLM to extract an entity graph from text, partitions it with the Leiden algorithm, and pre-summarizes every community. We explain how this answers global questions that vector RAG cannot, what the paper actually measured, and why indexing cost is the real trade-off.
A customer wanted their whole six-year support archive put to work. The system built on a filtered five percent answered better than the one built on all of it, because the archive contained superseded advice that reads exactly like current advice. What we filtered on, why recency alone was not enough, and how we made the discarded ninety-five percent a decision the customer could inspect rather than a black box.
Released on June 20, 2024, Claude 3.5 Sonnet beats Anthropic's flagship Claude 3 Opus at one fifth of the price and twice the speed. We examine the benchmark data, the collapsing cost-per-quality curve since GPT-4, and why Artifacts points toward working outputs instead of chat transcripts — limits included.
A survey counted well over fifty named prompting techniques. We ran the eleven that plausibly applied against one labelled set: two helped, one actively hurt, and the other eight were indistinguishable from noise on our tasks. Which two, why the technique that hurt is the one most often recommended, and the reason a catalogue of techniques is more useful as a list of things to test than as a list of things to adopt.
Ours abstained on two percent of questions and should have abstained on nine. Firmer instructions moved the number and moved it in both directions, refusing questions it could answer perfectly well. What worked was separating two different reasons for not knowing, giving each its own signal and its own response, and accepting that one of them cannot be detected from the model at all.
This 2024 guide shows how to build and maintain golden test sets for LLM products: sample production queries, label them with experts despite criteria drift, cover known failure modes, and calibrate automated judges against expert labels. Dataset maintenance — not tooling — is the real evaluation bottleneck. With concrete numbers and dated sources.
The delivery date was in the order and the agreed window in the contract, so no retrieved passage could contain both. The obvious answer is a knowledge graph over the corpus. We measured first: the link both documents needed was a contract number they already carried, and a deterministic join on a field beats a learned association whenever the identifier exists. Where that stops being true, and what we would build instead.
Announced on 13 May 2024, GPT-4o processes text, vision, and audio in a single end-to-end network. We examine the verified numbers — 320 ms average voice latency, half the price of GPT-4 Turbo, five times the rate limits — and separate what the API delivers today from what the launch demos promise.
Our agent opens pull requests and cannot merge them, runs tests and cannot change the pipeline. None of those limits came from a policy document; each came from a specific afternoon. What Yang et al. call the agent-computer interface is the part we spent most time on, and the rule underneath all of it: the tools define the blast radius, so an action that cannot be undone by a human in a minute does not get a tool.
This April 2024 guide puts parameter-efficient fine-tuning into practice. We cover how LoRA works, what QLoRA changes with 4-bit quantization, the current open-source toolchain, and a decision framework for tuning versus retrieval versus prompting — including what adapters do not deliver and how thousands of them run on one GPU.
A new architecture reached the general press and three customers asked in one week whether it changed our recommendations. The four questions we work through, why most exciting results are about a layer we do not touch, and the distinction that answers nearly all of these calls: a result can be genuinely important to the field and irrelevant to a system that is limited by its data.