The best LLM call in our pipeline was the one we deleted

The call summarized findings that were already correct

It summarized. When a review run fans out across a large tender, dozens of blocks finish independently, each returning structured findings with citations attached. A final LLM call wove them into one synthesis. That call double-counted budget figures, added latency to every run, and paraphrased findings that were already correct. We deleted it and now concatenate the raw results instead. The diff removed 119 lines and added 35.

Some context. The pipeline in question reads tender packages for construction and infrastructure projects: hundreds of pages of contract terms, technical descriptions and quantity schedules. A review run classifies the documents, splits them into blocks, and fans out dozens of agent runs in parallel. The deleted call sat at the very end, where all the parallel findings come back together, because it felt obvious that a model should merge them.

Nothing got worse when it went. The parallel agents already return their findings in a fixed structure, so placing them next to each other loses no information. The synthesis layer had no upside left to offer, only the standing risk of any model pass: every trip through an LLM is a chance for a number to change, and a changed number in a tender review is a defect, not a style choice.

The formatter we fired

We had met the pattern before. An earlier generation of the pipeline ended with an LLM that formatted aggregated research into the delivery structure. It was slow, it cost tokens, and now and then it got creative with content it was only supposed to arrange. Today that step is schema validation followed by JSON.stringify. Zero model calls, identical output, every time.

It sounds like a joke about model selection: the best model decision we made that year was JSON.stringify. But the reasoning is boring and solid. Formatting is transformation, transformation has a right answer, and work that has a right answer should not pass through a sampling process.

Majority voting killed recall, so it lasted one evening

Majority voting is the textbook defence against hallucination: run several replicas, keep what most of them agree on. We already run up to five replicas of our extraction agent over every chunk of a document, each with a different behavioural directive, so voting was an afternoon of work. We implemented it, diffed the results against real tender documents, and deleted it the same evening.

The diff explained why. Our replicas run at low temperature, and the exact matching upstream of them is deterministic, so hallucinated finds are already rare by the time a vote could be held. What the replicas actually disagree about are the rare valid finds only one of them catches: the requirement in an appendix that only the replica reading back to front reaches with attention to spare. A vote discards precisely those. Voting fights hallucination; when the hallucination is already dead upstream, the only thing it kills is recall.

It worked in the demo. We removed it anyway.

Not every subtraction was an internal step. We built extraction of drawing scales, the 1:50 or 1:100 printed on a drawing, so that quantities could be sanity-checked against geometry. It shipped, and it looked good in demos. We reverted it. Scale annotations sit in inconsistent places, often per viewport rather than per drawing, and a confidently wrong scale is worse than no scale in a tool contractors lean on while pricing bids. In our domain, "usually right" is a liability.

The feature can come back the day it can be verified deterministically. Until then, its absence is the safer feature.

LLMs for judgment, code for transformation

Four subtractions, one pattern. Each deleted call sat at a point in the pipeline where the task had a right answer, and each was replaced by something deterministic or by nothing at all. The method was the same every time: implement, diff against real documents, believe the diff.

The working rule that emerged: LLMs for judgment, code for transformation. A model earns its place where the task genuinely requires reading and weighing — does this clause shift risk onto the contractor, does this requirement apply to this bid. Merging results, formatting output and moving data between steps is transformation, and code does transformation faster, cheaper and identically on every run. Everything that can be deterministic should be.

The instinct after a strong demo is to add model calls. Our documents keep teaching us the opposite discipline. Every call we delete is one fewer place where a number can change on its way to an engineer who is going to rely on it.