QuoteBench shows coding-agent scores can conceal command-transport failures

AI software developmentAgent evaluationPreprint14 August 2026 arXiv listing

Authors: Shangao Li, Yao Zhang, Volker Tresp and Yuanyuan Yang. Status: preprint; submitted 13 August 17:57 UTC and listed 14 August.

Across 56 one-shot tasks, the authors report that adding an unescaped parser reduced success by 55.4–73.2 percentage points. Model adaptation recovered much of the loss for six configurations, while two did not recover.

Why it matters and impact: Coding-agent evaluations must record the generation contract, serialisation, shell path and final-state validator. A single benchmark score can hide fragile harness behaviour.