Engineering notes
Every article starts with a practical question, explains the necessary terminology, and links to primary sources. We separate external evidence, project experience, and conclusions — and state what cannot be inferred from them.
Adding a third-party tool server takes about a minute, which is the value of a standard interface and the reason nobody reviews what they add. A tool description is text the model reads and trusts, so it is an input channel as much as a capability. What we check before one runs, what we refuse outright, and why the credentials matter more than the code.
Vector storage was a rounding error at two hundred thousand chunks and a line item at four million, because it grows with the corpus rather than with usage. Quantising the vectors cut memory by roughly three quarters at a retrieval quality cost we could measure and accept. What we measured first, the two-stage arrangement we ended up with, and the cheaper fix we should have tried before any of it.
Manus launched in March 2025 with viral demos and state-of-the-art GAIA claims. The article compares the self-reported numbers with the first independent tests, explains what the benchmark does not measure, and argues why constrained agents with approval gates ship to production today while general autonomy does not — yet.
We stripped names, addresses and customer numbers before sending tickets to a hosted model, and a colleague identified the customer from what was left in about ten seconds. Direct identifiers are the easy half. What we changed once we accepted that redaction bounds exposure rather than removing it, where we stopped sending data at all, and why a placeholder that keeps its meaning is worth more than one that does not.
Customers who ask about fine-tuning mean the model that writes the answer. On the two projects where training on customer data actually paid, it was the retriever: far cheaper, far smaller, and aimed at the failure that was actually costing us, which was finding the wrong passage rather than writing about the right one badly. What the training data has to be, and the maintenance nobody mentions.
Eight customer systems, an afternoon each. None came out high risk, which was the boring part. The useful part was that writing down what each system does, what it decides and what a person still decides forced questions our design reviews had never asked, and two systems changed as a result. What we produce per system now, and why we say plainly that this is not legal advice.
Our early-2025 approach to LLM guardrails uses layered policies with known miss rates, PII redaction with Presidio, output filtering with Llama Guard 3 and moderation APIs, human approval gates for irreversible actions, and guardrails versioned as code with attack, benign, and regression test suites in CI.
The fourth time we wrote an adapter so a model could query an order system, it took two days and looked exactly like the previous three. A standard tool interface removed the duplication and moved the remaining work somewhere it belongs: deciding what a tool should expose. What we standardised, what we deliberately did not, and the security property that a shared interface makes easier to state and easier to get wrong.
We ran all nine steps of a pipeline against a model that reasons before answering. Two improved clearly, six were unchanged, and one got measurably worse. The pattern is not difficulty, it is whether a step has an intermediate decision at all. Also the thing customers most often get wrong: the visible reasoning reads like an explanation and is not one you can put in an audit trail.
Deep research agents spend minutes planning, browsing, and synthesizing instead of answering in one shot. We define the pattern, examine Google's Gemini Deep Research from December 2024, and quantify the weak point: citation quality. Measured baselines show why the verification duty stays with the reader — and what that means for engineering teams.
Two percent of extractions failed to parse and a retry fixed them, which looked like a solved problem. Constraining generation to the grammar removed the failures entirely and then showed what the retries had hidden: the documents that failed to parse were the documents the step was getting wrong. Where we constrain, where constraining makes things worse, and why a valid record is still not a correct one.
Classic RAG retrieves once and generates once. Agentic RAG instead lets the model decompose queries, retrieve iteratively, and critique its own context. We define the pattern, summarize the evidence from IRCoT, Self-RAG, and CRAG, state when a retrieval loop beats one-shot lookup, and show how to keep loop costs bounded.