IEEE Access · Vol. 14 · pp. 61784–61808

Open access · 2026

SimBench.

A framework for evaluating and diagnosing LLM‑based digital‑twin generation for multi‑physics simulation.

Jingquan Wang¹, Andrew Negrut², Hongyu Wang³, Harry Zhang¹, Dan Negrut¹

¹ Mechanical Engineering, University of Wisconsin–Madison · ² Computer Science, Rice University · ³ Electrical & Computer Engineering, University of Wisconsin–Madison

Paper arXiv Code Data BibTeX

Fig. 2 of the paper — a subset of the simulation scenarios in SimBench: one row per category, from gear trains to rovers on deformable terrain.

§ A Abstract

We introduce SimBench, a benchmark designed to evaluate the proficiency of simulator‑oriented LLMs (S‑LLMs) in generating digital twins (DTs) that can be used in simulators for virtual testing. Given a collection of S‑LLMs, this benchmark ranks them according to their ability to produce high‑quality DTs. We demonstrate this by comparing over 33 open‑ and closed‑source S‑LLMs.

Using multi‑turn interactions, SimBench employs an LLM‑as‑a‑judge (J‑LLM) that leverages both predefined rules and human‑in‑the‑loop guidance to assign scores for the DTs generated by the S‑LLM, thus providing a consistent and expert‑inspired evaluation protocol. The J‑LLM is specific to a simulator, and herein the proposed benchmarking approach is demonstrated in conjunction with the open‑source Chrono multi‑physics simulator. Chrono provided the backdrop used to assess an S‑LLM in relation to the latter’s ability to create digital twins for multibody dynamics, finite element analysis, vehicle dynamics, robotic dynamics, and sensor simulations. The proposed benchmarking principle is broadly applicable and enables the assessment of an S‑LLM’s ability to generate digital twins for other simulation packages, e.g., ANSYS, ABAQUS, OpenFOAM, StarCCM+, IsaacSim, and pyBullet.

§ 1 Framework

A judge that explains its score

Execution metrics such as Pass@k are all‑or‑nothing: one minor defect in a simulation script yields a zero, even when the digital twin is qualitatively close to correct. Similarity metrics such as CodeBLEU and ROUGE‑L reward surface overlap rather than physics. SimBench instead calibrates a judge LLM against expert scores — once, with a human in the loop — and then uses the frozen judge to deliver a 0–100 score plus itemized diagnostics that attribute every deduction to a concrete aspect of the generated script.

The SimBench pipeline. Multi-turn task prompts are sent to candidate LLMs, whose responses are compared with expert ground-truth responses. A human expert scores validation pairs, and the judge prompt is iteratively optimized on code with seeded errors until the judge's scores match the expert's, producing the calibrated SimBench Judge (J-LLM). — **Fig. 1 of the paper** — the SimBench pipeline. The judge is calibrated on validation pairs of ground‑truth and generated digital twins, its prompt iteratively optimized to match expert scores; the calibrated J‑LLM then evaluates each S‑LLM from the generated DT, the ground‑truth DT, and the API documentation.

J-LLM · Ref+Doc

Reference script and API documentation. The headline SimBench metric — rank correlation of ρ = 0.69 with human‑assessed Pass@1, the strongest correlate of Pass@1 among all the metrics compared.

J-LLM · Ref

Reference script only. Stricter and cheaper; still correlates at ρ = 0.57 with Pass@1 — ahead of Compile@1 (0.49), CodeBLEU (0.42) and ROUGE‑LSUM (0.15).

J-LLM · Doc

Documentation only. Without a reference, the judge drifts toward surface similarity (ρ = 0.94 with ROUGE‑LSUM) — evidence that the reference script is what grounds the score in physics.

Correlation matrix of seven metrics: Pass@1, Compile@1, three J-LLM variants, CodeBLEU, and ROUGE-LSUM. J-LLM with reference and documentation correlates 0.69 with Pass@1, the highest; CodeBLEU 0.42 and ROUGE-LSUM 0.15 correlate weakly; the documentation-only judge correlates 0.94 with ROUGE-LSUM. — **Fig. 4 of the paper** — correlation between metrics over the human‑assessed subset. Rubric judging with a reference tracks functional correctness (Pass@1) far better than compile checks or similarity scores.

§ 2 The benchmark

Thirty-four systems, five physics regimes

Every system ships with expert‑authored ground truth in PyChrono and a three‑turn dialogue that moves from a vague build request to a demanding extension. Tasks average 985 prompt tokens and 1,415 solution tokens — roughly 7.5× the prompt length of HumanEval and 61.6× that of MBPP.

102

turn-level tasks

physical systems

turns per system

S-LLMs evaluated

3,000+

expert-scored dialogues

rubric dimensions

Multibody systems

MBS

Rigid-body dynamics with joints, contact, collision and friction.

pendulumslider_crankgearmass_spring_damperparticles

Finite elements

FEA

Flexible beams, cables and shells; vibration, buckling, stress and strain.

beamcablebucklingtableclothrotor

Sensors

SEN

GPS, IMU, lidar and camera models attached to running simulations.

lidarcameragps_imuveh_app

Vehicle dynamics

VEH

Full vehicle models — driver, powertrain, tires — on rigid and deformable terrain.

hmmwvcitybussedanm113gatorkrazmanfedauazbusscmscm_hillrigid_highwayrigid_multipatchesart

Robotics

RBT

Rovers and mobile robots — including NASA’s VIPER and Curiosity — plus ROS bridges and deformable terrain (SCM).

vipercuriosityturtlebotvehrossensroshandler

Chips carry the dataset’s own system identifiers — e.g. viper is NASA’s VIPER lunar rover, scm is deformable soil‑contact terrain, hmmwv is the HMMWV military truck.

Three turns, rising difficulty

Turn 1 · vague request

Build from scratch

Construct the digital twin from an abstract, minimally specified description — an ~87‑token prompt and nothing else.

mean score across models 20.2 / 100

Turn 2 · sharp request

Modify & debug

Meet concrete requirements for changing the turn‑1 script — a ~1.4k‑token prompt carrying the full previous code.

mean gain +29.3 · 87.6% of cases improve

Turn 3 · sharp request

Extend the scenario

A challenging extension demanding multi‑step reasoning and coordinated changes across components.

mean change −6.2 · 22 of 34 systems decline

What the judge scores

Additive deductions from 100 across six weighted dimensions — e.g. −15 for each missing essential component, −15 for API misuse that changes simulation behavior. The judge returns an itemized critique and a machine‑parsable final score.

Completeness40

Correctness30

Code quality10

Efficiency10

Error handling & robustness5

Visualization5

Points at stake per dimension, out of 100.

§ 3 Results

Nobody clears fifty

Across 33 open‑ and closed‑source models, the best overall SimBench score is 49 out of 100. Reasoning models hold the top three positions, open‑weights models sit within a few points of the frontier — and the headroom above every bar is the point: generating a working digital twin from a natural‑language request is far from solved.

SimBench leaderboard — top 11 of 33

1claude-4-sonnet49

2o346

3claude-3.7-sonnet43

4o4-mini42

4qwen3-235b-a22bopen42

6gemini-2.5-pro41

6gpt-4.1-mini41

6gpt-4o-mini41

6llama-4-maverickopen41

6llama-4-scoutopen41

6llama-3.3-70bopen41

J‑LLM Ref+Doc score (0–100), macro-averaged over all 102 turn-level tasks — Table V of the paper. Bars are drawn against the full 0–100 scale. Scores across all 33 models range down to 21 (phi‑3‑medium‑128k).

Table view — all three judge modalities

#	Model	Ref+Doc	Ref	Doc	Weights
1	claude-4-sonnet	49	39	55	proprietary
2	o3	46	37	67	proprietary
3	claude-3.7-sonnet	43	36	55	proprietary
4	o4-mini	42	35	54	proprietary
4	qwen3-235b-a22b	42	32	55	open
6	gemini-2.5-pro	41	31	62	proprietary
6	gpt-4.1-mini	41	34	55	proprietary
6	gpt-4o-mini	41	28	49	proprietary
6	llama-4-maverick	41	33	44	open
6	llama-4-scout	41	34	38	open
6	llama-3.3-70b	41	34	42	open
…	21 further models	≤40
33	phi-3-medium-128k	21	20	20	open

Bubble scatter plot of average judge score against model release date from early 2024 to mid 2025, colored by provider, with a positive fitted trend line (r equals 0.624, p equals 0.0001). Recent reasoning models such as o3 and claude-4-sonnet sit at the top right; phi-3 models sit at the bottom left. — **Fig. 3 of the paper** — temporal evolution of S‑LLM performance. Average of the three judge modalities against release date: a steady climb (ρ = 0.624), with reasoning models leading the frontier.

49 / 100

The ceiling is low

The best overall score in the study, from claude‑4‑sonnet. Digital‑twin generation remains an open problem.

13%

Few scripts truly run

Best Pass@1 in the human‑assessed 19‑model subset: even the strongest model produces a functionally passing script about once in eight attempts.

+29.3

Code context is decisive

Mean turn‑2 gain once the model sees its own previous script plus sharp requirements — 87.6% of the 1,122 model–system pairs improve.

22 / 34

Extensions break twins

Systems whose scores drop again at turn 3 (mean −6.2), where the request demands multi‑step, cross‑component changes.

31.3

Sensors are hardest

Lowest category average; the single hardest system is lidar at 23.4. FEA is the most tractable category at 41.0.

ρ = 0.62

Models are improving

Correlation between release date and the average judge score across the three modalities (p < 0.001). Reasoning‑enabled models hold the top three positions.

§ 4 Get started

Score your own model

The repository ships the full benchmark — 34 systems × 3 turns with ground truth — plus the calibrated judge as both a CLI and a library. The judge defaults to gpt‑4o‑mini; any OpenAI‑compatible provider works via --base-url.

shell · evaluate & reproduce

# environment
$ conda env create -f environment.yml
$ conda activate chronobench
$ export OPENAI_API_KEY=sk-...

# score your agent (34 systems × 3 turns)
$ python -m chronobench.score <your-agent> \
      --responses-dir runs --dry-run
$ python -m chronobench.score <your-agent> \
      --responses-dir runs

# reproduce the published evaluation
$ bash scripts/fetch_published_data.sh
$ python scoring/rank_llm.py

python · the judge as a library

from chronobench.judge import evaluate_script

ev = evaluate_script(
    candidate_code,
    reference=truth_code,
    api_doc=open("api/api.txt").read(),
)

print(ev.score)      # 0–100
print(ev.rationale)  # itemized deductions —
                     # feed them back to your agent

The main branch moves beyond the paper (leaner dependencies, test suite). For the exact published pipeline and data, check out tag paper-ieee-access-2026; the frozen run itself is archived on Zenodo.

§ 5 Beyond Chrono

A simulator-agnostic recipe

SimBench is demonstrated on Chrono, but the benchmarking principle is portable: given an expert dataset for another ecosystem, the same workflow yields a calibrated judge for it. Porting means re‑instantiating three interface points.

i · task specification

What does the simulator consume?

Recast the multi‑turn protocol around the native artifact — Python scripts, input decks, XML/URDF/SDF models, or plugin code.

ii · context sources

What grounds the judge?

API documentation plus expert reference solutions, used to calibrate the judge for the new ecosystem before freezing it.

iii · execution oracle

What counts as running?

Platform‑specific compile and pass checks — an Abaqus job that submits, a MuJoCo rollout with no NaNs and bounded trajectory error.

ANSYSABAQUSOpenFOAMSTAR-CCM+Isaac SimPyBulletMuJoCoGazebo

Companion benchmark — PyChronoBench

280 auto‑gradable multiple‑choice questions probing fine‑grained PyChrono API knowledge — contact modeling, body creation, solver settings, sensor configuration. A fast complement to SimBench’s full generation tasks.

PyChronoBench

§ 6 Citation

Cite this work

@article{simbench2026,
  author  = {Wang, Jingquan and Negrut, Andrew and Wang, Hongyu
             and Zhang, Harry and Negrut, Dan},
  journal = {IEEE Access},
  title   = {SimBench: A Framework for Evaluating and Diagnosing
             LLM-Based Digital-Twin Generation for Multi-Physics
             Simulation},
  year    = {2026},
  volume  = {14},
  pages   = {61784-61808},
  doi     = {10.1109/ACCESS.2026.3685519}
}

IEEE Xplore arXiv preprint Dataset archive