Research Summary: The Copy Ceiling: An Input-Exposure Control for Ontology-Grounded Generation over Curated Corpora

Research details a method to ground language models in curated ontology corpora, which significantly improved target-name recall from 0.265 to approximately 0.92 across ten models. However, this performance remains below a 'copy baseline' of 0.964, indicating that current evaluation metrics for recall may not adequately distinguish between reasoned answers and verbatim copies of input context. The study highlights limitations in assessing genuine reasoning in ontology-grounded generation.

Why it matters

This research provides critical insights into the evaluation of artificial intelligence models, particularly in natural language generation and knowledge grounding. It underscores the limitations of recall metrics in assessing true reasoning versus superficial reproduction of input, which is vital for developing robust and genuinely intelligent systems. Understanding these evaluation nuances is essential for setting realistic expectations and guiding future AI development.

Key insights

• A node was developed to ground a replaceable language model in a maintained ontology corpus.
• Grounding increased target-name recall from 0.265 (unaided) to approximately 0.92 across ten models.
• A 'copy baseline' (verbatim copying of context) achieved a recall score of 0.964.
• All tested grounded models performed 0.022 to 0.067 below the copy baseline.
• The study indicates that high recall scores, particularly in the presence of contextual information, do not necessarily confirm the presence of reasoning, as copied information can achieve similar scores.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.24885