When AI Is Your Pastor: A Benchmark for Theological Triage and Pastoral Guidance in Large Language Models

Large Language Models (LLMs) are increasingly consulted for counsel on matters of faith, doctrine, and pastoral care. Traditional benchmarks do not adequately assess LLMs' performance in these complex, nuanced domains, which often involve core beliefs, inter-denominational disagreements, prudential judgments, and critical safety considerations. A new benchmark, FMG-Bench, has been introduced to specifically evaluate LLMs in English-language Christian theological triage and pastoral guidance contexts across 120 scenarios and 14 advanced models.

• Individuals are using Large Language Models (LLMs) for complex questions related to faith, doctrine, and pastoral care.
• These inquiries extend beyond standard information retrieval, encompassing nuanced theological and ethical considerations.
• Current AI evaluation benchmarks lack the capacity to assess LLMs' performance in scenarios requiring theological triage, discerning core beliefs, acknowledging disagreements, exercising prudential judgment, or addressing safety-critical pastoral situations.
• The Faith & Moral Guidance Benchmark (FMG-Bench) has been developed to address this gap.
• FMG-Bench v1 comprises 120 scenarios designed to evaluate LLMs in English-language Christian theological and pastoral guidance.

Reported by arXiv — Computers and Society · International. This briefing summarises the publication for executive use; the document itself is not reproduced here.