AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents

Research has identified a novel vulnerability, 'AgentHijack,' demonstrating the feasibility of visual patch attacks to execute command injection against computer-use agents (CUAs). These attacks leverage visual patches embedded on webpages to manipulate CUAs through their screenshot input, vision-language model (VLM) generation, action parsing, and environment execution stages, potentially leading to verifiable environmental consequences. The study evaluated these attacks across five GUI-agent or VLM backends, achieving varying rates of successful compromise.

This research highlights a significant and emergent cybersecurity vulnerability in the rapidly evolving domain of AI-driven computer-use agents. The ability to inject commands visually could undermine the integrity and security of automated systems and sensitive data, posing a new vector for cyber threats. Understanding and mitigating these visual patch attacks is critical for maintaining trust in AI and autonomous systems, and for protecting digital infrastructure from novel exploitation methods.

A new framework, 'AgentHijack,' allows for end-to-end evaluation of image-triggered command injection against computer-use agents (CUAs).

Forward consideration, not a verified fact.