Benchmark cases
17 matched
Problem
What the build needed to solve
My contribution
Built the schema/unit loader, independent calculation and answer-extraction paths, domain checks, failure taxonomy, provenance-tracked capture harness, Markdown audit report, and CI gate.
LLMs produce plausible-looking mechanical-engineering calculations with unit slips, wrong governing formulas, invalid thin-wall assumptions, arithmetic errors, and correct answers reached by incorrect reasoning. A useful verifier must recompute the physics independently and be explicit about the narrow domain it actually covers.
Constraints
Design boundaries
Recompute supported quantities independently rather than trusting the model's arithmetic.
Treat every benchmark case as a hash-verified capture with recorded provenance (elicited versus organic).
Report only within the covered domain: thin-wall pressure vessels, axial stress, cantilever controls, and finite-width stress concentration.
Verifier regressions must fail CI against the committed annotation set.
Build log
Design evolution
Iterations, issues, and fixes documented as a compact notebook timeline.
Revision
Failure mode
Design change
Result
Failure taxonomy
“Wrong answer” is too coarse to test a verifier against.
Defined FM-01 units, FM-02A/B formula and assumption, FM-03 arithmetic, FM-04 stress concentration, and FM-07 right-answer/wrong-reasoning.
Each case annotates specific expected failure modes.
Independent recomputation
Trusting the model's stated number only re-grades its own claim.
Recompute mechanics from the inputs and compare against the model's extracted reported answer.
Detected modes are computed by the checker, not read from the model.
Provenance and CI gate
Elicited and organic captures are different evidence classes.
Hash-verified captures with recorded provenance and a detected-equals-expected gate in CI.
17 of 17 committed cases match; 2 skipped.
Testing
Results
Failed cases
0
Skipped
2
Models covered
5
Failure families
6
Evidence
Elicited captures
Each committed case pairs an LLM calculation capture with an annotation of expected failure modes; CI passes only when MechAudit's independently computed detected modes equal that annotation.
The current domain covers thin-wall pressure vessels, axial stress, cantilever controls, and finite-width stress concentration across five models.
Scope note: Detected modes match annotations on committed elicited captures. This measures the verifier against a bounded benchmark, not overall model capability, and does not yet include organic “wild” failures captured during real engineering work.
Reflection
Engineering lessons
- A failure taxonomy makes verification a checkable detection task instead of a vague judgment.
- Recomputing the physics independently is the only honest way to catch a correct-looking wrong answer.
- Elicited and organic captures are different evidence classes and must be labeled as such.
Gallery
CAD, fabrication, and test views
Benchmark result
Detected failure modes matched the annotation on all 17 committed cases (elicited captures across five models; two skipped).
Links