Engineering notes
Every article starts with a practical question, explains the necessary terminology, and links to primary sources. We separate external evidence, project experience, and conclusions — and state what cannot be inferred from them.
Four hundred model-generated test cases made our suite look thorough and measure almost nothing: they agreed with the system because they came from the same understanding of the task. Where model-generated data genuinely earns its place, the line we draw between generating an input and generating the expected answer, and what a verifier has to be for self-training to mean anything.
Four failure classes define this January 2024 RAG taxonomy: retrieval misses, stale indexes, the lost-in-the-middle effect, and citation drift. We define each mode, cite the measurements behind it, and argue for evaluating retrieval and generation as separate systems — recall@k on one side, faithfulness metrics on the other.
Every few months a customer asks whether they should run the model themselves, and almost never because of capability. Open weights reaching parity with hosted models changed a procurement question, not an engineering one. The four things we weigh, what self-hosting actually costs once someone has to be on call for it, and the two situations where we recommend it without hesitating.
This January 2024 survey compares JSON mode from OpenAI's DevDay 2023, function calling as a schema contract, grammar-constrained decoding with Outlines and llama.cpp, and validation-first pipelines on Pydantic v2. It shows what each method guarantees, what none of them guarantee, and why structure beats parsing prose.
Reference pipelines show eight or nine stages. Ours has four, and two exist only because a specific failure forced them in. A survey published that December gave names to what we had assembled by accident, which made it possible to say what we had skipped on purpose. What each stage cost, what it caught, the one we added and removed again, and the two we still do not run.
On 6 December 2023, Google announced Gemini 1.0 in three sizes: Ultra, Pro and Nano. We separate delivered capability from marketing — benchmark methodology and the edited demo video — and examine what a second frontier-class vendor means for builders: routing policies, provider-neutral interfaces and task-level evals instead of single-vendor bets.
A customer asked us to automate a step their staff finish in forty seconds. We declined, because short does not mean simple: the task drew on context that exists nowhere in the systems an assistant can reach. The three tests we run before agreeing to automate anything, and why our refusal list has grown rather than shrunk.
Pure vector search fails on part numbers, error codes, and other exact identifiers. We explain the structural reason, define BM25 and dense retrieval precisely, show how Reciprocal Rank Fusion combines both without score normalization, and review measured gains from hybrid retrieval and early rerankers such as Cohere Rerank — with numbers from 2023 benchmarks.
Our system prompt reached nine hundred words assembled by six people over four months, every line added for a reason and no reason written down. Nobody would delete anything because nobody could predict the cost. How we made prompts reviewable, what a test per instruction changed, and the two hundred words that turned out to do nothing.
At DevDay on November 6, 2023, OpenAI announced GPT-4 Turbo with a 128k context window, the Assistants API with Retrieval and Code Interpreter, and GPTs. We summarize the verified numbers, the new API primitives, and the concrete limits — and state what remains your responsibility: evaluation, data governance, cost control, and vendor risk.
Our assistant contradicted itself about a delivery date it had confirmed six minutes earlier. The instinct is to reach for a bigger window; what actually fixed it was writing decisions into a store the moment they are made, and reading them back as facts rather than as history. What we page in, what we never page in, and the failure that made the rule non-negotiable.
Demos and gut feeling are not an evaluation strategy for LLM applications. We cover golden test sets, model-graded evaluation with GPT-4 as judge, the position and verbosity biases quantified by Zheng et al. in 2023, and a layered evaluation stack that runs in CI — including what each technique does not measure.