Friday, April 27, 2018

Data Replication and Sociocultural Change

In the gray area between science and pseudoscience.

By continuing to focus on the necessity for replication, psychological science misses an important and all-pervasive psychological phenomenon: the impact of social and cultural change on behavior. Or put otherwise, our discipline misinterprets failure to replicate behavioral results if we do not consider that social and cultural change can produce systematic shifts in behavior. Data on the connection between social change and behavioral change point to a new role for "replication": not to show that results can be duplicated, but to reveal behavioral effects of sociodemographic and cultural change in the intervening years between original and replicated procedure, whether those be surveys, standardized behavioral procedures, or intelligence tests.
The argument is fine as far as it goes, but it begs several questions.  If psychometric data cannot be effectively replicated because of “social and cultural change,” what are we to do then?  Accept single point data as definitive for a given period of time, and accept that changes in findings are definitely associated with changes in society and culture?  How do we distinguish sociocultural-driven data change from experimental error, or faulty design, or fraud, or whatever?  And if “social and cultural change” is really that pervasive and rapid as to make replication of results (replication attempts are typically, but not always, performed not very long after the initial study), then what is that telling us?  Is this change good?  Dangerous?  Do we know what the long-term impact is going to be?  And why view such change as some sort of inevitable “force of nature” (very subtly implied, intentionally or not, in Greenfield’s comments), when much of this change is being forced upon us by (typically self-interested) group agendas?