How Does LGBTQIA+ Identity Affect LLM Behavior? Implications for Requirements Engineering of Mental Health AI Systems

A recent research study investigated how Large Language Models (LLMs) respond to mental health questions when LGBTQIA+ identity is disclosed, compared to no disclosure or straight identity disclosure. The study, using ChatGPT and questions from a mental health repository, suggests that explicit LGBTQIA+ identity disclosure influences LLM-generated responses in mental health contexts. This raises concerns regarding fairness and equitable support for vulnerable populations within AI-driven healthcare systems.

Why it matters

The deployment of AI systems, particularly LLMs, in sensitive domains like mental healthcare necessitates a robust understanding of their behavior towards diverse user groups. Ensuring equitable and unbiased support for all populations, especially those considered vulnerable, is critical for maintaining public trust and ethical standards in AI development and application.

Key insights

• Large Language Models (LLMs) are being integrated into healthcare and mental health support systems.
• Concerns exist regarding fairness toward vulnerable populations, specifically LGBTQIA+ individuals, within these systems.
• Limited empirical research has explored the impact of explicit LGBTQIA+ identity disclosure on LLM responses.
• The study analyzed 450 ChatGPT responses across three prompt conditions: no identity disclosure, explicit straight identity disclosure, and explicit LGBTQIA+ identity disclosure.
• Findings indicate that explicit LGBTQIA+ identity disclosure affects LLM-generated responses in mental health contexts.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.00352