A Finger on the Scale: research examines interference in AI benchmarks

A research preprint posted on arXiv on September 2 by Jiarui Li and seven co-authors examines how reusable 'skills' given to AI agents can be manipulated to tilt benchmark results, a problem the authors call Skill Policy Integrity.

The paper describes a covert 'SkillShift' attack, in which a subtly modified skill steers an agent toward attacker-favoured selections. The authors report attacker-favoured selection rates of 81.33 per cent and 63.33 per cent in their test settings.

The findings come with caveats: this is a preprint that has not completed peer review, and the results come from constructed test settings rather than observed real-world attacks.

The broader point stands regardless. Reusable skills, the shared building blocks that give AI agents their abilities, should be audited as supply-chain policy artefacts, the way software dependencies already are, because a compromised skill quietly compromises every agent that loads it.

Organisations adopting skill-based agent frameworks should track where each skill comes from, pin versions, and review updates before deployment rather than pulling them automatically.