Police Psychology:  Reproducibility

by Gary S. Aumiller, Ph.D.  ABPP

So, my 11-year old daughter had to do a science project earlier this year and she decided because her mother was constantly saying that playing on the IPad was hurting her, she would test out whether IPad play really had an impact on her attention span.  So, we set it up with three alternate forms of a neurological test, the Trail Making Test, a connect the dots type of activity, (okay I helped a little), one before, one in the middle and one at the end.  There was an hour and a half of Minecraft in-between each of the trials.  We got her friends over and measured the change from trial to trial, with both time and number of errors as the variables.  The results….well, I’ll get to that.

We start training kids from an early age on the rigors of scientific method and how to make a scientific study.  We train them how to test a theory and how to make a hypothesis.  We train them that science requires experiments to answer questions and learn more about our world.  What I think we forget to train them in is that one study does not provide a definitive answer but only a suggestion.  The ignorance of the statement that it is only a suggestion is how we come up with a bunch of “Fake Science” being reported and guiding our way of life.

Okay, I borrowed “Fake Science” from “Fake News” ‘cause it would make a good weekly TV show.  That is sort of the problem also — good studies make better entertainment or as one minister once told me “don’t let the truth interfere with a good story.”  In a study by Brian Nosek called the Reproducibility Project, Dr. Nosek and a team of 269 co-authors started trying to reproduce studies from three prestigious journals in psychology.  In the Reproducibility Project, as it has come to be known, only 39% of 100 studies from 98 papers in the journals were reproducible.  Meaning when they redid 100 studies only 39 got the same results in the same direction.  That’s less than a third of a chance that the study you read is correct.  Meanwhile, people were relying on these studies and making decisions on the results of these studies.  But, how can we even be sure this replication study is for real.

A Stanford epidemiologist, which is a person that studies disease origins, says that 39% number is high.  In fact the number should be closer to 80% of studies are not reproducible, in other words, you have 1 in 5 chance of having accurate information.  He said the Reproducibility Project tested studies only from distinguished journals.  If you move to all  journals, the data is even more favorably skewed, and further the amount of fraud and misapplied statistics is ridiculous.  There are major incentives for studies to find results in a certain direction.  For one, you must get it published.  Second, the news channels will cover it if it is profound, and thirdly, funding comes to those that get results.  In all, only 3% of research money goes to replicate that which is already reported.  Once it is published, people forget about caring whether it is accurate.  So much for double checking your work that we are all taught in elementary school.

Now for the really bad news!  This seems to apply to more sciences than just psychology.  In cancer biology, one analysis showed that only 6 of 53 studies are replicable.  THAT’S A WHOPPING 11%!!  In other words, if your doctor is depending on the current journal articles to select a treatment for your cancer, 89% percent of the time they will give you the wrong treatment.  An overall study of medical treatments out early this year said from The Journal of the American Medical Association, The Lancet, and The New England Journal of Medicine (three of the world’s most prestigious medical journals) only 34% of the treatments that were found to be efficacious in original studies were actually efficacious in replication.  You got a one third chance of it working.  That’s a good batting average, but not the odds I am looking for from my doctor.

Where does this lead us.  Hell, I don’t know.  You are sort of at a physician’s mercy, I guess.  Read up when you have something wrong with you.  Find the things that have worked and been replicated.  When something’s gone awry, try something else.  There are some major individual differences in people and remember that when you pick treatments.  What works for you may not be what works for other people.  Look at studies with a little more caution.  Don’t get sold on one technique until you see a ton of replication and definitely check the connection of the researcher to the treatment.  And finally keep in mind — replicate, replicate, replicate.

So, what did we find with my daughter’s science project?  Between the first and second test there was the beginning of deterioration suggesting time on the IPad could be bad news.  The scores on the second test were longer in time it took to complete the test and slightly higher in errors.  Right before the final test though, pizzas had arrived, so there were tons of errors, but they were definitely much faster completing the tests.  So, I would guess a scientist would say the IPad increases mental processing, but you are not as efficient.  Or something like that!  I suspect someone might get a different response in replication.  :}

Please share this article from down below.

Please join the email list on the top of the sidebar and you can get these sent to your email.  Also follow me on Twitter for other articles and ideas, and YouTube at https://www.youtube.com/channel/UCfjNw0510ipr3bX587IvAHg .

Feel free to donate if you like the site.