Small Foundation Models of Human Cognition and Behaviour

Research from arXiv explores the efficiency of 'Small Foundation Models of Human Cognition and Behaviour' by training fourteen models, ranging from 135 million to 14 billion parameters, on the Psych-101 dataset, comprising 10.7 million trial-level choices across 160 experiments. The study indicates that for in-distribution data, model scale has minimal impact, with models from 0.6 billion to 1 billion parameters performing comparably to a 70 billion parameter baseline. However, for out-of-distribution generalization to novel tasks, larger models demonstrate a clear advantage, suggesting a steeper scaling gradient in these scenarios.

Why it matters

This research provides insights into the optimal scaling of models designed to simulate human cognition, indicating that smaller models can be highly effective for specific tasks. This has implications for resource allocation and development strategies in artificial intelligence, highlighting a trade-off between model size, computational cost, and generalizability.

Key insights

• Small foundation models (0.6B to 1B parameters) can achieve performance comparable to much larger models (70B parameters) when fine-tuned on human behavioral data for in-distribution tasks.
• The performance of models on in-distribution data shows a narrow band, suggesting a ceiling effect where increased scale offers diminishing returns.
• For out-of-distribution generalization to novel tasks, a steeper scaling gradient is observed, with larger models demonstrating clear advantages.
• The study questions whether large language models fine-tuned on human behavioral data process task structure or exploit statistical shortcuts.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.05224