Research Summary: Understanding AI Provider Recommendations in Local Service Markets

A research audit of AI assistant recommendations for local service providers, such as doctors or financial advisors, indicates significant issues with factual accuracy. When generating referrals without web search capabilities, both open-weight and proprietary AI models frequently fabricate recommendations, especially in domains where web coverage is limited. This poses substantial risks to users relying on these systems for critical service selections.

Why it matters

The findings highlight a critical challenge in the reliability and trustworthiness of AI-generated information, particularly for high-stakes decisions like selecting professional services. This directly impacts user safety, regulatory compliance, and the reputation of AI platforms and the organizations that deploy or endorse them.

Key insights

• AI provider recommendations were audited across four registry-backed service domains in the 100 largest U.S. metropolitan areas.
• Recommendations were matched against official registries (e.g., Medicare clinician/facility records, SEC advisor disclosures).
• Three conditions were tested: an open-weight model, a proprietary model without web search, and the proprietary model with web search.
• Without web search, both AI models (open-weight and proprietary) largely fabricated recommendations, particularly where domain information was sparsely covered online.
• Only 4% of the open-weight model's recommended doctors and 11% of the proprietary model's recommendations matched a clinician in the queried city when operating without web search.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.18341