Small AI Models Mimic Human Cognition Effectively.

The use of large language models (LLMs) fine-tuned on human behavioral data as "cognitive proxies" has gained traction, but questions remain regarding the necessary scale and whether these models truly understand task structure or merely exploit statistical shortcuts. A recent study investigated this by training fourteen models, ranging from 135 million to 14 billion parameters, across four architectural families. These models were trained on Psych-101, a dataset comprising 10.7 million trial-level choices from 160 psychological experiments. The findings revealed that for in-distribution tasks, model scale had surprisingly little impact; models between 0.6 billion and 1 billion parameters performed comparably to a 70 billion parameter baseline. This suggests that a relatively small model size is sufficient to capture the nuances of human behavior within familiar experimental paradigms. However, when tested on out-of-distribution tasks with novel structures, a clear scaling gradient emerged, with larger models demonstrating a distinct advantage in generalization. To understand the information these models utilize, diagnostics were performed. By progressively stripping prompt channels (instructions, stimuli, feedback, choice history) and permuting trial order, researchers found that masking stimuli and feedback content significantly degraded performance, indicating that models rely heavily on these elements rather than just choice history. Furthermore, permutation tests showed invariance on independent trials but sensitivity where trial order was crucial, confirming that these models process task structure. The study concludes that small, cognitively fine-tuned models hold promise for estimating noise ceilings in psychological experiments, though their generalization scope is currently limited by the training paradigms.