Research Summary: When AI Is Your Pastor: A Benchmark for Theological Triage and Pastoral Guidance in Large Language Models

Large Language Models (LLMs) are increasingly consulted for counsel on matters of faith, doctrine, and pastoral care. Traditional benchmarks do not adequately assess LLMs' performance in these complex, nuanced domains, which often involve core beliefs, inter-denominational disagreements, prudential judgments, and critical safety considerations. A new benchmark, FMG-Bench, has been introduced to specifically evaluate LLMs in English-language Christian theological triage and pastoral guidance contexts across 120 scenarios and 14 advanced models.

Why it matters

The expanding use of AI for sensitive guidance like faith and pastoral care presents significant implications for trust, ethics, and societal influence. Understanding and benchmarking AI capabilities in these domains is crucial for developing responsible AI, mitigating potential harms, and informing policy around AI's role in personal and spiritual development. This research highlights an emerging area where AI's impact extends beyond traditional technical or commercial applications.

Key insights

• Individuals are using Large Language Models (LLMs) for complex questions related to faith, doctrine, and pastoral care.
• These inquiries extend beyond standard information retrieval, encompassing nuanced theological and ethical considerations.
• Current AI evaluation benchmarks lack the capacity to assess LLMs' performance in scenarios requiring theological triage, discerning core beliefs, acknowledging disagreements, exercising prudential judgment, or addressing safety-critical pastoral situations.
• The Faith & Moral Guidance Benchmark (FMG-Bench) has been developed to address this gap.
• FMG-Bench v1 comprises 120 scenarios designed to evaluate LLMs in English-language Christian theological and pastoral guidance.
• The benchmark has been applied to evaluate 14 advanced LLMs.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.12324