AgentHijack: Visual Patch Attacks on Multimodal Computer-Use Agents

Research has identified a novel vulnerability, 'AgentHijack,' demonstrating the feasibility of visual patch attacks to execute command injection against computer-use agents (CUAs). These attacks leverage visual patches embedded on webpages to manipulate CUAs through their screenshot input, vision-language model (VLM) generation, action parsing, and environment execution stages, potentially leading to verifiable environmental consequences. The study evaluated these attacks across five GUI-agent or VLM backends, achieving varying rates of successful compromise.

Why it matters

This research highlights a significant and emergent cybersecurity vulnerability in the rapidly evolving domain of AI-driven computer-use agents. The ability to inject commands visually could undermine the integrity and security of automated systems and sensitive data, posing a new vector for cyber threats. Understanding and mitigating these visual patch attacks is critical for maintaining trust in AI and autonomous systems, and for protecting digital infrastructure from novel exploitation methods.

Key insights

• A new framework, 'AgentHijack,' allows for end-to-end evaluation of image-triggered command injection against computer-use agents (CUAs).
• Visual patches, embedded on web pages like GitHub Pages or CSDN clones, can induce verifiable environmental consequences through CUAs.
• The attack targets the full operational chain of CUAs, including screenshot input, VLM generation, action parsing, and environment execution.
• Experiments across five open-source or publicly available GUI-agent or VLM backends aggregated 600 online cases.
• The study reported a Target Attack Success Rate (T-ASR) of 84.5%, a Target Attack Persistence Rate (TAPR) of 47.0%, and an End-to-End Attack Success Rate (E2E-ASR) of 20.3%.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.09212