Research Summary: terms.txt: A Consent and Compensation Protocol for Agentic Web Access

The traditional unwritten agreement governing web access, where sites allowed crawlers in exchange for traffic from search engines, is faltering due to the rise of AI crawlers and agents. These automated clients now dominate web requests, with AI platforms fetching significant volumes of data without proportional return traffic. Existing control mechanisms like robots.txt are insufficient to express identity, purpose, terms, or pricing for machine access, and can be circumvented. A new standard, terms.txt, is proposed to address these challenges by enabling path-specific, purpose-driven machine access terms, alongside a secure, origin-enforced negotiation protocol for consent and compensation.

Why it matters

This development highlights a fundamental shift in how digital information and resources are accessed and valued online, posing significant governance and operational challenges for organizations relying on web presence. Addressing the imbalance of data extraction by automated agents without adequate compensation or clear consent is critical for maintaining the economic viability and integrity of web-based operations and content creation. Establishing clear terms for machine access can protect intellectual property and ensure equitable value exchange in the evolving digital ecosystem.

Key insights

• The unwritten bargain of the open web, where sites allowed crawlers in exchange for return visitors, is being undermined by AI crawlers and agents.
• Automated clients now constitute the majority of web requests, with AI training dominating classified crawling activity.
• Major AI platforms are observed to fetch thousands of pages for each visitor they return, indicating an imbalanced exchange.
• The current robots.txt protocol is inadequate for modern machine access, lacking capabilities for expressing identity, purpose, terms, or pricing.
• robots.txt is also prone to circumvention, and newer alternatives are often proprietary CDN features.
• The research proposes terms.txt, a new protocol for per-path, per-purpose machine access terms.
• terms.txt would be complemented by an origin-enforced exchange mechanism utilizing Web Bot Auth signatures, signed intent, delegation tokens, and HTTP 402 negotiation.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.11152