Research Summary: Context Is Not Free: Carbon-Aware LLM Routing for Multi-Turn LLM Web Services across Geo-Distributed Data Centers

Research indicates that carbon-aware routing for large language model (LLM) web services across geo-distributed data centers faces trade-offs when handling stateful, multi-turn conversations. While routing requests to regions with lower carbon intensity or smaller models can lower emissions, moving ongoing conversations requires recomputing localized key-value (KV) context caches, which can generate more carbon emissions than the transfer saves.

Why it matters

This analysis matters because well-intentioned sustainability and workload-routing strategies can inadvertently increase overall carbon footprints if technical statefulness is overlooked. Systemic infrastructure efficiency requires factoring hidden computational overheads into digital decarbonization roadmaps. Organizations scaling AI services must align operational routing models with actual net-emissions impacts rather than point-in-time regional metrics.

Key insights

• LLM web services run across geo-distributed data centers where grid carbon intensity fluctuates over time and geography.
• Multi-turn conversational LLM services are stateful, relying on key-value (KV) caches tied to the specific deployment hosting earlier turns.
• Rerouting an active conversation to a cleaner region necessitates recomputing conversation history, incurring carbon costs that increase as conversations lengthen.
• Context recomputation costs can negate or exceed emissions reductions gained from moving execution to greener regions.
• Existing carbon-aware routing models treat multi-turn conversational queries as independent, stateless interactions, undermining their environmental efficacy.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.35230