Researchers identify severe paraphrase fragility in vision-language robot reward models
A study introducing ROBORMBENCH reveals that minor phrasing changes can flip VLM progress evaluations of identical robot behavior.
Original cover art, generated for this story. THE VISSION does not republish third-party press imagery.
- Researchers have published a study detailing 'paraphrase fragility' in Vision-Language Models (VLMs) used as reward functions.
- The team introduced ROBORMBENCH, containing 2,390 real-robot trajectories and 21,673 verified paraphrased prompts.
- The findings show that scaling model size or using Chain-of-Thought reasoning does not reliably mitigate this reward instability.
A research team spanning Yonsei University, Seoul National University, and Carnegie Mellon University has published a paper revealing a critical vulnerability in the deployment of Vision-Language Models (VLMs) as reward functions for robotic learning. The study, titled 'Same Trajectory, Contradictory Rewards,' establishes that current general-purpose VLMs exhibit severe 'paraphrase fragility'—meaning they assign drastically different rewards to the same physical robot trajectory based purely on how the instruction is phrased.
VLM-based reward models require absolute paraphrase invariance to be safe and reliable: instructions like 'grasp the red block' and 'pick up the crimson cube' should yield identical progress rewards for the same behavior. However, the researchers demonstrated that minor syntactic changes or lexical substitutions frequently cause VLMs to flip their evaluations entirely, marking a successful robot action as a failure under a paraphrased goal description, or generating highly unstable training signals.
To systematically measure this instability, the authors introduced ROBORMBENCH, a new evaluation framework consisting of 2,390 real-world robot trajectories with ground-truth progress labels, evaluated against 21,673 verified paraphrased commands. The benchmark exposes that paraphrase-induced reward errors are prevalent across both proprietary and open-source models, and that standard mitigations—such as scaling up model size or utilizing explicit Chain-of-Thought prompts—fail to reliably stabilize reward outputs.
The paper concludes that general-purpose VLMs are currently too unstable to serve as autonomous robotic reward functions without specialized training. Instead, the researchers demonstrate that reward models trained with trajectory-grounded supervision exhibit significantly greater paraphrase invariance. They propose new robustness metrics, including the Score Crossing Rate (SCR) and Flip Rate (FR), to help future roboticists audit and safeguard AI-driven reward pipelines against linguistic drift.
This research exposes a critical safety hazard in the transition toward autonomous robotic training. If vision-language reward models cannot maintain linguistic consistency, reinforcement learning agents will learn to exploit phrasing bugs rather than master physical manipulation tasks, creating severe compliance risks. This study forces roboticists to abandon simple off-the-shelf VLM prompts in favor of rigorous, trajectory-grounded reward architectures.
Will roboticists adopt ROBORMBENCH's Flip Rate metrics to benchmark and secure multi-modal reward functions before training physical manipulators?
Still open. When the paper finds out, it will say so here and on the open questions page — including if it got this wrong.