ai
SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models
- Source
- arXiv — Computers and Society
- Published
- Last verified
- 18 Aug 2026
- Confidence
- High
- Evidence
- Original document retained
- Reading time
- 1 min
- Country
- International
- Relevant to
- Research & Evidence, Operations & Delivery, Technology & Data
Executive summary
What happened, and why should leadership care?
Research indicates a significant disparity in safety performance for large language models (LLMs) when transitioning from English to low-resource languages. An evaluation of four open-weight instruction-tuned models revealed substantial 'refusal gaps' in their ability to decline harmful-intent prompts in Somali compared to English, highlighting a critical limitation in current global AI safety evaluations.
Why this matters
Why is this strategically important?
This research reveals a critical technical and ethical challenge in the global deployment of artificial intelligence. It underscores that current AI safety measures are not uniformly effective across different linguistic contexts, potentially exposing users in low-resource language communities to disproportionate risks from harmful AI outputs. Addressing these 'refusal gaps' is vital for ensuring equitable and secure AI development and deployment worldwide.
Key insights
What should be noted from the evidence?
- Current LLM safety evaluations are predominantly English-centric, leading to insufficient assessment of low-resource languages.
- A study using SomaliBench v0, a benchmark of 100 harmful-intent prompts in both English and Somali, evaluated four open-weight instruction-tuned models.
- Models assessed include Llama-3.1-8B-Instruct, Gemma-2-9B-Instruct, Qwen-2.5-7B-Instruct, and Aya-23-8B.
- All four models exhibited large English-to-Somali refusal gaps, ranging from 0.40 to 0.93, indicating a higher likelihood of generating undesirable outputs in Somali.
- These refusal gaps were statistically significant, confirmed by paired bootstrap and exact McNemar tests.
Evidence and confidence
How far can this assessment be trusted?
High confidence. Named institution, original document retained and analysis corroborated.
Analysis is prepared editorially by Aziz Shuaib Ausi. The original publication remains the authoritative record, and executive judgement remains entirely human.
Source
Where does this originate?
Reported by arXiv — Computers and Society · International. This briefing summarises the publication for executive use; the document itself is not reproduced here.
Read the original publication