Research Summary: Activating the Annotator: Operationalising Lived Experience in Data Annotation

A research study introduces AXLE, a method designed to activate annotators' lived experiences in subjective natural language processing tasks. Moving beyond standard demographic questionnaires that treat positionality as a static covariate, the framework employs pre-annotation scenario vignettes and conversational micro-narratives to dynamically prime annotator perspectives before labelling commences.

Why it matters

This research addresses a core methodological flaw in how subjective machine learning training data is collected and contextualised. By establishing ways to actively prime lived experience rather than relying on passive demographic surveys, institutions can achieve more faithful representations of human judgment in artificial intelligence models. This matters because model alignment and fairness initiatives risk systematic failure if the underlying annotator ground truth misrepresents human perspective.

Key insights

• Perspectivist natural language processing currently lacks operational methods to effectively integrate annotator positionality into data labelling.
• Standard industry practice treats annotator positionality as a static demographic trait collected once at recruitment and modelled as a covariate.
• Current annotation approaches rely on the unverified assumption that demographic background remains actively engaged during task execution.
• The AXLE framework introduces an interactive pre-annotation stage utilizing scenario vignettes and axis-guided discussions.
• AXLE compiles these guided interactions into first-person micro-narratives returned to annotators to prime relevant lived experiences prior to labelling.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.12701