Engineering notes
Every article starts with a practical question, explains the necessary terminology, and links to primary sources. We separate external evidence, project experience, and conclusions — and state what cannot be inferred from them.
We put a 7B model in as a stopgap for one classification step and never replaced it. Not because small models caught up, but because we finally scoped the step so that its difficulty matched the model. What that scoping looks like, where we still reach for a large model without hesitating, and the measurement that settles the argument.
On August 22, 2023, OpenAI opened GPT-3.5 Turbo fine-tuning. We define what fine-tuning actually improves — output format, tone, narrow tasks — and what it does not: adding knowledge. Using the October 2023 price list we calculate training cost, the eightfold inference premium and the break-even, and show why retrieval-augmented generation remains the correct channel for facts.
Retrieving on every turn made our assistant worse at the easy questions: three loosely related excerpts and a hedge where one sentence would do. Where we put the decision to retrieve, why we kept it in the pipeline rather than training it into the model, and the two signals that turned out to predict it well enough.
Autonomous agents surged in spring 2023: AutoGPT passed 74,000 GitHub stars within weeks, BabyAGI fit into 140 lines of Python. Six months later, few run in production. We trace the ReAct pattern behind the hype, quantify runaway loops and real costs, and describe the constrained tool-use loop that works: budgets, schemas, checkpoints.
For six weeks our faithfulness score climbed while support tickets climbed with it. Automated RAG evaluation was not broken; it was answering a narrower question than we thought we had asked. What reference-free metrics genuinely catch, the failure they are blind to by construction, and the cheap human check we now run underneath them.
Prompt injection remains unsolved one year after Simon Willison named the vulnerability class. We define direct and indirect injection, explain why retrieval-augmented generation and tool use widen the attack surface, review the mitigations available in September 2023, and state plainly what each one does not do. No complete fix exists.
A customer wanted a fine-tuned model for facts that changed weekly. We built retrieval instead, and explained why a training run is a snapshot with a shelf life. The rule we use to decide between the two, the two cases where tuning genuinely is the right answer, and what the cheap-tuning era changed about the conversation.
Chunking determines what a document QA system can retrieve at all. We contrast fixed-size and recursive splitting, quantify chunk size and overlap trade-offs against the context windows of 2023, and explain why metadata is mandatory. With verified numbers from Pinecone, OpenAI and the Lost in the Middle paper — plus an outlook from September 2023.
Past thirty tools our agent began choosing plausibly wrong ones, and no amount of prompt work fixed it. Cutting to nine did. Why we scope toolboxes by task rather than by system, what that costs in routing, and why the research direction of teaching models thousands of APIs points somewhere different from where most products need to go.
Vector-database companies raised more than 160 million dollars in spring 2023 alone. We explain what approximate nearest neighbor search actually does, how the HNSW index works, and why Postgres with the newly released pgvector 0.5.0 is often sufficient — plus the concrete criteria that justify a dedicated system.
Two agents talking to each other solved the task in the demo and became undebuggable in production: no step to point at, no state to inspect, no way to answer why it failed. What we kept from the conversational pattern, what we replaced with an explicit state machine, and the one case where we still let agents negotiate freely.
An agent with good public benchmark numbers failed on the third step of a five-step process, in a way no benchmark had asked about. Why we treat agent benchmarks as a filter rather than a decision, what a journey test looks like instead, and why the simulated counterpart is the part that actually pays.