What the best AI models leave out of an engineering report

A wrong design value reads exactly like a right one

An engineering report sizes things. A geotechnical design memo decides the excavation method and the slope ratio. A stormwater study sizes the detention basin. A vibration risk analysis sets the limit values that protect the neighbouring buildings during blasting. An error in any of them is expensive because it is hard to see: a wrong design value inside fluent, internally consistent prose reads exactly like a right one.

General language models now write good engineering prose and pull data out of source material with striking accuracy. So the useful question is narrower. Can a system produce a draft that a reviewing engineer would accept: with everything the report type requires, with preliminary dimensions committed, and with references that hold when someone checks them with the standard in hand?

Six report types, with the delivered report as answer key

We took six report types from everyday civil engineering practice. For each one the reference was a real report, delivered by an established consultancy in a municipal or national transport project in Sweden. The system under test received what the original consultant had: the project brief and the source material, such as the ground investigation report, field and laboratory protocols and CPT evaluations. It never saw the delivered report.

From each delivered report we built a checklist of 27 to 33 control points: structure, classifications, assessments per area, calculations and the control process. A point is a hit, a partial or a miss, and a partial counts as half. Correctness is judged against the source material, not blindly against the reference. Where a draft departs from the delivered report and the sources support the draft, it scores as a hit. Every cited value was checked against the sources and every calculation chain was recomputed.

Yesper was compared with GPT-6 Astra and Claude Opus 5.5, the current frontier models from OpenAI and Anthropic. Both ran at medium reasoning effort with the identical brief and source text, and a plain instruction to write the report without asking questions. Each draft is a single generation. None was re-sampled.

97 percent against 83 and 75

Yesper reached a mean of 97 percent agreement with the delivered reports. Claude Opus 5.5 reached 83 percent and GPT-6 Astra 75. Yesper led on all six report types, by 5 to 26 points over the better of the two.

The margin follows what the report type demands. It is smallest where the task is closest to faithful compilation: 5 points on the ground investigation report. It is largest where the report has to commit derived values: 26 points on the vibration risk analysis, and 15 on both the geotechnical design memo and the contaminated land risk assessment.

Counted together, the six checklists hold 184 control points. Yesper hit 174, met 10 in part and missed none. Opus 5.5 hit 130 and missed 7. Astra hit 110 and missed 16.

Neither general model invented source data. Spot checks matched 23 of 23 values in Astra's geotechnical memo, and Opus 5.5 reproduced 540 of 540 laboratory cells. The data was right. What was missing came after the data.

Two ways to fall short

GPT-6 Astra invents nothing because it does not decide. Its drafts give no slope ratio, no safety class, no vibration limit values derived from the standard's factors and no pre-development flow, and they write "cannot be excluded" where an assessment is due. Each of the client's questions is answered with the investigation that would be needed to answer it. That is a reviewer's checklist. The client ordered a consultant's judgement.

Claude Opus 5.5 commits, and its numbers hold. In the vibration risk analysis every factor comes from the standard's own tables and all 42 values in the calculation chains recompute. Then the content a reviewing engineer looks for next is missing. The risk analysis has no section on sensitive equipment, no noise table and no air-blast limits. The memo has no soil profile per street and no stability estimate with numbers. The stormwater study has no detention time. The site investigation has no list of governing documents.

The clearest view is the part of the vibration risk analysis that carries its liability: the limit value each neighbouring building and installation receives, derived from ground conditions, building class, material and distance under SS 4604866 and SS 02 52 11. Of the nine control points in that block, Astra hit none, Opus 5.5 hit six and Yesper hit all nine, with every chain shown and every factor sourced.

Beyond the point score, each draft faced one yes-or-no question: does its normative basis survive a check with the standard in hand? Yesper passed on all six report types. Each general model passed on four.

A false source label is worse than a missing one. In Opus 5.5's environmental site investigation, eight guideline cells sit inside an otherwise flawless classification table with a wrong value or a handbook reference that does not hold. No sample changed class. But a reviewer with the handbook open cannot tell which cells to trust, and the table looks auditable.

More reasoning effort does not close it

The obvious objection is that the general models were not thinking hard enough. We tested it on the previous Claude generation, Opus 5, by running it at each of the five reasoning-effort levels the API offers, with prompt and inputs unchanged, on the geotechnical design memo and the vibration risk analysis. On the memo, generation time rose with every level, from just over three minutes to more than ten. Quality did not follow. Four of the five levels landed between 69 and 72 percent and the highest reached 78. On the vibration risk analysis the series ran 69, 72, 69, 77 and 69.

On the vibration risk analysis every level failed the audit, and for the same reason: factor values invented inside otherwise genuine calculation chains. Each level corrected one or two constants from the level below and invented new ones. That is drawing again from the same memory. It is not deeper reasoning. Elsewhere the same model attributed four soil-reuse guideline values to Naturvårdsverket's Handbook 2010:1. They do not appear in it.

The newer model did what effort could not. Opus 5.5 at medium effort matched Opus 5's highest-effort tally on the memo in half the time, and passed the vibration audit with the standard's own factors. Then it stopped at the same boundary: 15 points short of Yesper on the memo and 26 on the risk analysis.

One example from the stormwater study shows why the check has to sit outside the model. The design rain intensity for a 20-year rain of 20 minutes is 190 litres per second and hectare under Dahlström's formula, the one Svenskt Vatten's P110 recommends. Opus 5 wrote out the correct formula, entered the return period in years where the formula takes months, got 84.5, and then did flawless arithmetic on that number. The detention basin came out 63 percent too small by the method the draft itself declared. Yesper checks constants like this one against a control value. The two current models both got it right.

Three things to ask of any AI-written report

The distance that remains is not model intelligence. It is the procedure of each report type: what the report must contain, which values it must commit and how they are derived, and which sources govern it. Yesper works to such a procedure for each report type. Structure and required content come from sources that can be cited, not from the model's memory: the standards, the geotechnical industry's application documents, Naturvårdsverket's guidance, Svenskt Vatten's P110 and more.

• Does it commit? A draft that answers each question with the investigation needed to answer it has handed the engineering back to you.
• Is the derivation on the page? A limit value without its factor chain cannot be checked. Neither can a guideline value marked "per the handbook" from memory.
• Is everything the report type requires there? Fluent prose hides a missing safety class or control programme better than a rough draft does.

A fourth question follows from the first three: what does the system do when an input is missing? Yesper lowers the confidence of the wording, lists the supplementary investigations that are needed and closes every report with a log of its assumptions. Where the delivered report rests on a software run, such as a slope stability factor, Yesper does not invent the result. It specifies the calculation for the engineer's own run, the way a signing consultant would.

Our own benchmark, and what it does not measure

We designed this test and we scored it, using separate evaluation agents that had no access to the system that wrote the drafts. No outside party has audited it yet. Each general-model draft is one generation, so variation between runs is not measured. Two of the six report types have so far been run against one project each, the other four against two projects with opposite profiles. And the score measures content: formatting, drawings and company templates are outside it for all three systems. The full report, with tallies per report type, is available on request.

One condition favoured the general models. They received the source material as extracted text. On a real assignment it arrives as A1 drawings, sounding diagrams and some 200 pages of appendices with handwritten field protocols. Yesper reads that material directly, and on a run from the raw appendices it scored 30 of 30 control points. The report has no general-model column for that run, because the material cannot be pasted into a prompt.

An independent benchmark points the same way. In AEC-Bench, 196 tasks from architecture, engineering and construction published by the AI company Nomic in March 2026, the best score on submittal review was 23.1 percent. The tasks and the method differ from ours. The pattern is the same: a general model, a regulated engineering deliverable, and a result that stops short.