When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts

A research audit of Vision-Language Models (VLMs) acting as 'judges' for image selection revealed significant reliability and bias concerns. The audited VLM, despite its parameter count, demonstrated performance barely exceeding random selection and falling short of a basic similarity baseline. Key issues identified include a strong bias towards selecting the first image presented and high sensitivity to the order in which candidate images are displayed, indicating a lack of robust decision-making consistency.

This research highlights critical limitations in the reliability and consistency of current VLM technology when deployed as decision-makers in image selection processes. Organizations relying on or considering the use of VLMs for content moderation, selection, or ranking must account for inherent biases and order-dependency, which can lead to unpredictable outcomes and undermine user experience or operational integrity.

VLM judges tasked with image selection demonstrated performance only marginally better than random chance when evaluated against independent human ratings.

Forward consideration, not a verified fact.