terms.txt: A Consent and Compensation Protocol for Agentic Web Access

The traditional unwritten agreement governing web access, where sites allowed crawlers in exchange for traffic from search engines, is faltering due to the rise of AI crawlers and agents. These automated clients now dominate web requests, with AI platforms fetching significant volumes of data without proportional return traffic. Existing control mechanisms like `robots.txt` are insufficient to express identity, purpose, terms, or pricing for machine access, and can be circumvented. A new standard, `terms.txt`, is proposed to address these challenges by enabling path-specific, purpose-driven machine access terms, alongside a secure, origin-enforced negotiation protocol for consent and compensation.

This development highlights a fundamental shift in how digital information and resources are accessed and valued online, posing significant governance and operational challenges for organizations relying on web presence. Addressing the imbalance of data extraction by automated agents without adequate compensation or clear consent is critical for maintaining the economic viability and integrity of web-based operations and content creation. Establishing clear terms for machine access can protect intellectual property and ensure equitable value exchange in the evolving digital ecosystem.

The unwritten bargain of the open web, where sites allowed crawlers in exchange for return visitors, is being undermined by AI crawlers and agents.

Forward consideration, not a verified fact.