Research Summary: What Counts as Evidence? Epistemic Cue Utilization in Large Language Models

A research study evaluating 13 large language models across 500 scenarios demonstrates that models exhibit wide divergence in how they weigh conflicting evidence. While models demonstrate near-identical baseline ability to recognize source reliability, their actual decisions range from negligible shifts to complete choice reversals when source histories are exchanged, revealing distinct underlying dependencies on source history, evidence lineage, temporal cues, or presentation order.

Why it matters

This research demonstrates that an artificial intelligence model's ability to identify credible sources does not guarantee sound or consistent reasoning when evaluating conflicting inputs. As organizations deploy autonomous systems to process ambiguous intelligence, unverified cue weighting can cause decisions to hinge arbitrarily on formatting or presentation order rather than factual validity. Addressing these epistemic inconsistencies is essential to establishing dependable automated analysis and risk management.

Key insights

• Across 13 large language models, identical recognition of source reliability leads to inconsistent outcomes, ranging from negligible influence to complete choice reversal when source histories are swapped.
• Models demonstrate distinct behavioral profiles: some prioritize source history, whereas others are predominantly influenced by evidence lineage, event time, or presentation sequence.
• The evaluation framework utilizes a 500-scenario benchmark spanning five tracks to isolate cue identification, cue influence, and competition between source, temporal, and presentation signals.
• Interventions applied to three Qwen model checkpoints examined cue manipulation via matched sham material, fact restatement, and structured evidence frameworks.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.12788