QuoteBench shows command-transport failures that coding-agent scores hide

A preprint by Shangao Li, Yao Zhang, Volker Tresp and Yuanyuan Yang, submitted August 13 at 17:57 UTC and listed August 14, 2026, examines how coding-agent evaluations can go wrong in the harness rather than the model.

Across 56 one-shot tasks, adding an unescaped parser reduced measured success by 55.4 to 73.2 percentage points, the authors report. Adaptation recovered most of the loss in six configurations; two did not recover.

Why it matters: coding-agent evaluations should specify the generation contract, serialisation, shell path and final-state validator. A single score can hide fragile harness behaviour.