Small Foundation Models of Human Cognition and Behaviour

Research from arXiv explores the efficiency of 'Small Foundation Models of Human Cognition and Behaviour' by training fourteen models, ranging from 135 million to 14 billion parameters, on the Psych-101 dataset, comprising 10.7 million trial-level choices across 160 experiments. The study indicates that for in-distribution data, model scale has minimal impact, with models from 0.6 billion to 1 billion parameters performing comparably to a 70 billion parameter baseline. However, for out-of-distribution generalization to novel tasks, larger models demonstrate a clear advantage, suggesting a steeper scaling gradient in these scenarios.

This research provides insights into the optimal scaling of models designed to simulate human cognition, indicating that smaller models can be highly effective for specific tasks. This has implications for resource allocation and development strategies in artificial intelligence, highlighting a trade-off between model size, computational cost, and generalizability.

Small foundation models (0.6B to 1B parameters) can achieve performance comparable to much larger models (70B parameters) when fine-tuned on human behavioral data for in-distribution tasks.

Forward consideration, not a verified fact.