Publication detail

In his 1956 APA Presidential Address, Lee Cronbach called for a merging of differential and experimental psychology. One main component was the use true experiments from experimental psychology to study individual differences. True experiments had control conditions, and used statistical contrasts to define effects that were free of nuisance variation. Cronbach reasoned that such experimentally defined effects would have greater construct validity than raw performance scores. Cronbach's merger has been repeatedly attempted, but the results have been lackluster, especially for measuring individual differences in cognitive control. Here we show through simulation that the merger is difficult because experimentally-defined contrasts are too noise-prone to be useful at the individual level. As a consequence, it is difficult to uncover even simple latent structures such as clusters or factors. To explore the merger, we provide a new measure of task goodness that is invariant across experiments with different numbers of people and, most importantly, trials. This new measure is a signal-to-noise ratio; how variable people are relative to how variable noise is in repeated trials. We survey the literature to establish typical SNR levels, which are then used in simulations. Latent cluster or factor structures were only recoverable in the largest of experiments comprising hundreds of people each observing hundreds of trials in each condition for each task, and only for the simplest structures. These disappointing results serve as a warning. Although Cronbach's merger is a great idea in theory, it faces substantial hurdles in practice.

@article{rouder_etal:preprint:experiments,
    title   = {{O}n {C}ronbach’s merger: {W}hy experiments may not be suitable for measuring individual differences},
    author  = {Rouder, Jeffrey and Chávez De la Peña, Adriana F. and Mehrvarz, Mahbod and Vandekerckhove, Joachim},
    year    = {preprint},
    journal = {PsyArXiv}
}