PHOTO ILLUSTRATION: THE GLOBE AND MAIL. SOURCE IMAGES: ISTOCK
Susan Pinker is a psychologist and columnist whose most recent book, The Village Effect, explores the science of social interaction.
By May of last year, the number of research papers on COVID-19 was doubling every two weeks. This steady churn has unleashed more than 200,000 journal articles on the coronavirus so far, more than 30,000 of them as preprints, meaning studies that are not yet peer-reviewed. Most are also too fresh to have been replicated by other scientists.
Still, many of these findings are cited by scientists and journalists alike and have been shared with millions of people. Indeed, the human brain seems to be the perfect growing medium for untested, often too-good-to-be-true ideas.
Story continues below advertisement
Just one example: A year ago, a preprint, followed by a published scientific paper, reported that an anti-parasitic drug, Ivermectin, used to treat river blindness in sub-Saharan Africa and head lice everywhere else, could suppress SARS-CoV-2. If this were true, it would be a godsend, as Ivermectin is cheap, FDA-approved as an anti-parasitic, and widely available.
The Ivermectin study did not succeed when replicated, however, and theres no clear evidence that it has any effect on humans with the disease. Like hydroxychloroquine a malaria drug that was touted as a cure by Donald Trump when he was the U.S. president Ivermectin has no clinical effects on COVID-19 in the real world.
But the cat video is out of the bag, so to speak. Ivermectin has become an internet meme; dozens of so-called scientific talks have received millions of views on YouTube, each one promoting Ivermectins anti-COVID benefits, with the backstory that pharmaceutical companies are suppressing this information lest it staunch the flow of profits from vaccines. In a short time, a single, non-replicated study has become the new bleach, or in medical history terms, the new snake oil.
Over the past 18 months, two pandemics have collided. On one hand, the SARS-CoV-2 virus was novel, terrifying and constantly evolving; there was a push for quick answers. On the other hand, science and medicine were in the throes of a replication crisis.
Many iconic studies, especially in my own field, psychology, were being repeated by other scientists and found to be wanting; their findings could not be duplicated. In other words, science was in the process of examining itself when COVID-19 struck. There was a desperate thirst for information and a glut of new studies. But there was also little time or patience for a basic due diligence step replication, simply repeating an experiment to see whether it produces the same results.
Now that COVID-19 infection rates are slowing down, its a good time to step back and look at what types of studies seem impossible to reproduce yet have remarkable staying power. Once launched, they continue to breathe new life into what are essentially rumours; they promote misinformation while giving it the patina of science.
Many of us think that science progresses in a straight line. But it zigzags. One hypothesis emerges, changing our outlook for a moment; if it fails to be confirmed by further evidence, it drops out of sight and another one takes its place. Onward and upward, in infinitesimal steps.
Recently, though, the fact-finding trajectory seems to have changed course. Attention-grabbing studies are briskly published. The press and other researchers latch on; a TED talk and a book contract often follow. Social media help turn it into a meme, one that is often surprising, easy to grasp and sticky, like an ear-worm or gossip. If the information hits a sensitive spot, such as a latent fear of hypodermic needles, contamination or public speaking, and also presents a tidy way to dispatch those anxieties, so much the better.
Somewhere along the way, though, another research team has repeated the experiment and couldnt get the same results. But a non-finding is usually non-newsworthy, and in any case, the surprising fact has already made a dent on our collective psyches. Thats just one way splashy findings like Ivermectin as a cure for COVID-19 continue to get attention, even after having been debunked.
Now, a fascinating new study out of University of California San Diego, led by behavioural economists Uri Gneezy and Marta Serra-Garcia, shows that experiments that could not be repeated have a bigger influence over time than the ones that could. In other words, the more interesting and novel-sounding the idea, the more it is cited by other scientists and the media, and the less likely it can be replicated.
There seems to be a trade-off between the wow factor of a study and its ability to be reproduced which should affect its credibility but does not. In Nature and Science, two high-profile and high-status journals, the non-replicable papers were cited 300 times more than the replicable ones.
Published in May in the journal Science Advances, the Gneezy-Serra-Garcia study analyzed the findings from three massive replication projects, two of them led by psychology professor Brian Nosek, from the Center for Open Science at the University of Virginia, and the third directed by behavioural economist Colin Camerer at the California Institute of Technology. All three focused on social science experiments that have been published in highly reputable journals. Volunteer scientists from the same field then repeated the experiment and hoped to get the same results, much as a cookbook author might test her recipes by baking the same torte in a different kitchen, with different utensils, to see if it looked and tasted the same.
Story continues below advertisement
To test a studys replicability alongside its popularity, Prof. Gneezy and Prof. Serra-Garcia matched each original studys outcomes to its citations on Google Scholar, starting roughly in 2008 and ending in 2019. Although they didnt cherry-pick the wow studies, they still found that just 39 per cent of psychology studies replicated, as did 61 per cent in economics, and 62 per cent of those published in Nature and Science.
Yet over a decade, studies that failed to replicate were cited 16 times more often a year than reproducible ones, with no significant change after it was shown that they couldnt be replicated. Papers that failed to replicate were cited much more than papers that were reproducible, Prof. Gneezy told me. If citations are just a proxy for how sexy an idea is, then the findings that are more interesting and get the most attention are the least likely to be true.
Some of the social science findings that havent been replicated include one showing that gripping a pen between your teeth, thus forcing a smile, makes you feel happy. Only one out of 18 attempts by other labs could reproduce this effect (it only works if no one is watching, apparently). Not just facial expression but body posture, too, is supposed to elicit emotions, according to a well-known study of power-posing. The idea that striking a victory stance legs braced, arms in a V can boost ones confidence and alter ones hormone levels (not to mention attenuate ones jitters before public speaking or a job interview) has become the poster child for the replication crisis, mainly because it was such an easy fix for the universal fear of failure. Even if the power-posing study didnt replicate, it continues to top the charts in scientific citations; a TED talk on the topic has garnered 61 million views.
The notion of stereotype threat which is a fancy way of saying that believing stereotypes about your gender, race or ethnic group becomes self-fulfilling prophecy has also largely failed to replicate. For example, the belief that they are bad at math necessarily hampers girls math performance has been studied extensively and is now widely accepted. Yet systematic replications of the phenomenon cant reproduce it. Still, stereotype threat has influenced educational policy, training programs and even admission practices at some universities.
Studies on growth mindset, implicit bias research and ego depletion have faltered in replications thus showing their findings to be exquisitely sensitive to context and statistical nuance. To extend the cooking metaphor, these findings might be like souffls that only rise when the weather is right. Or, they might simply be false. Either way, they persist in the scientific literature, in corporate and educational trends, and in the public imagination.
Story continues below advertisement
The Gneezy-Serra-Garcia study focused on social science. But the phenomenon of non-replicability has hit the worlds of natural science, economics and medicine as well. Heres just a small sampling: A paper published in the journal Nature last year reported that a small inhibitory molecule could tamp down the formation of tau, a sign of Alzheimers disease. Though the study could not be reproduced, it has been cited 605 times in papers published by other scientists. Similarly, animal studies showing successful treatment of Type 1 diabetes based on combining two existing drugs, a neurotransmitter and a malaria medication, raised hopes when published in the journal Cell in 2017. Two other prestigious teams tried and failed to reproduce the results. Still, the research has been cited 238 times.
Other findings that didnt pass muster include a 21 percentage-point boost for the soccer player who kicks first in penalty shots, published in 2010 in American Economic Review. The finding couldnt be coaxed out of a larger sample two years later. Yet the first-mover advantage has been cited 483 times in research, not to mention innumerable soccer broadcasts.
Why is this happening? The researchers who authored the Science Advances study speculate that journal editors perhaps unconsciously might overlook methodological problems in studies with intuitive appeal. These findings may be surprising, or they might provide an easy fix to a complex problem. Ivermectin for COVID-19, power-posing, stereotype threat and first-mover advantage in soccer all fit both bills. Once published, such findings take on a life of their own, gaining currency and momentum. Extraordinary claims require extraordinary evidence, Carl Sagan, the American astronomist, said. When our expectations are high, the burden of proof should be high, as well.
Still, its likely an overstatement to call this a crisis. Enthusiasm for replication and attempts to confirm catchy findings have been growing, and thats a good thing. Having studied the problem for years, Prof. Nosek, the executive director of the Center for Open Science, seems unconcerned.
This is about the scientific community self-scrutinizing, he said. Its not surprising that the things that push the boundaries fail to replicate. Thats how it should be. We should be trying things that are not likely to be true. Then we investigate to see if they are.
Keep your Opinions sharp and informed. Get the Opinion newsletter. Sign up today.
View post:
- Yes, But. The Annotated Atlantic. - November 7th, 2009 [November 7th, 2009]
- Health Insurance Benefit Costs by Region - November 8th, 2009 [November 8th, 2009]
- For an Operator, Please Press... - November 8th, 2009 [November 8th, 2009]
- Pollyanna With a Pen: Maine Governor Signs 18 New Health Care Bills into Law - November 8th, 2009 [November 8th, 2009]
- AMA Sounds the Alarm, Medicare Making Yet Another Attempt to Cut Reimbursement - November 8th, 2009 [November 8th, 2009]
- Mass Governor Asks Blue Cross to Keep Higher Employer Contribution - November 8th, 2009 [November 8th, 2009]
- Lifespan and Care New England Plan Monopoly (Again) - November 8th, 2009 [November 8th, 2009]
- Dirigo Health: Con Artists, Liars, and Thieves? - November 8th, 2009 [November 8th, 2009]
- New Orleans: Health Challenges - November 8th, 2009 [November 8th, 2009]
- August a Flurry of Activity - November 8th, 2009 [November 8th, 2009]
- Maine's Dirigo Health Savings One-Third of Original Estimate - November 8th, 2009 [November 8th, 2009]
- “Methodolatry”: My new favorite term for one of the shortcomings of evidence-based medicine - November 8th, 2009 [November 8th, 2009]
- Suzanne Somers’ Knockout: Dangerous misinformation about cancer (part 1) - November 8th, 2009 [November 8th, 2009]
- A science-based blog about GMO - November 8th, 2009 [November 8th, 2009]
- A Not-So-Split Decision - November 8th, 2009 [November 8th, 2009]
- Military Medicine in Iraq - November 8th, 2009 [November 8th, 2009]
- The effective wordsmithing of Amy Wallace - November 8th, 2009 [November 8th, 2009]
- A Science Lesson from a Homeopath and Behavioral Optometrist - November 8th, 2009 [November 8th, 2009]
- Join CFI in opposing funding mandates for quackery in health care reform - November 8th, 2009 [November 8th, 2009]
- Mainstreaming Science-Based Medicine: A Novel Approach - November 8th, 2009 [November 8th, 2009]
- Those who live in glass houses… - November 8th, 2009 [November 8th, 2009]
- J.B. Handley of the anti-vaccine group Generation Rescue: Misogynistic attacks on journalists who champion science - November 8th, 2009 [November 8th, 2009]
- When homeopaths attack medicine and physics - November 8th, 2009 [November 8th, 2009]
- The cancer screening kerfuffle erupts again: “Rethinking” screening for breast and prostate cancer - November 8th, 2009 [November 8th, 2009]
- All Medicines Are Poison! - November 8th, 2009 [November 8th, 2009]
- When Loud Wins: Will Your Tax Dollars Pay For Prayer? - November 8th, 2009 [November 8th, 2009]
- It’s All in Your Head - November 8th, 2009 [November 8th, 2009]
- The Skeptical O.B. joins the Science-Based Medicine crew - November 8th, 2009 [November 8th, 2009]
- The Tragic Death Toll of Homebirth - November 8th, 2009 [November 8th, 2009]
- What’s the right C-section rate? Higher than you think. - November 8th, 2009 [November 8th, 2009]
- Recombinant Human Antithrombin – Milking Nanny Goats for Big Bucks - December 13th, 2009 [December 13th, 2009]
- Does C-section increase the rate of neonatal death? - December 13th, 2009 [December 13th, 2009]
- Man in Coma 23 Years – Is He Really Conscious? - December 13th, 2009 [December 13th, 2009]
- Why Universal Hepatitis B Vaccination Isn’t Quite Universal - December 13th, 2009 [December 13th, 2009]
- Ontario naturopathic prescribing proposal is bad medicine - December 13th, 2009 [December 13th, 2009]
- Naturopaths and the anti-vaccine movement: Hijacking the law in service of pseudoscience - December 13th, 2009 [December 13th, 2009]
- The Institute for Science in Medicine enters the health care reform fray - December 13th, 2009 [December 13th, 2009]
- Neti pots – Ancient Ayurvedic Treatment Validated by Scientific Evidence - December 13th, 2009 [December 13th, 2009]
- Early Intervention for Autism - December 13th, 2009 [December 13th, 2009]
- A temporary reprieve from legislative madness - December 13th, 2009 [December 13th, 2009]
- A critique of the leading study of American homebirth - December 13th, 2009 [December 13th, 2009]
- Lose those holiday pounds - December 13th, 2009 [December 13th, 2009]
- Endocrine disruptors—the one true cause? - December 13th, 2009 [December 13th, 2009]
- Acupuncture for Chronic Prostatitis/Chronic Pelvic Pain Syndrome - December 13th, 2009 [December 13th, 2009]
- Evidence in Medicine: Experimental Studies - December 13th, 2009 [December 13th, 2009]
- Midwives and the assault on scientific evidence - December 13th, 2009 [December 13th, 2009]
- The Mammogram Post-Mortem - December 13th, 2009 [December 13th, 2009]
- An Influenza Recap: The End of the Second Wave - December 13th, 2009 [December 13th, 2009]
- The End of Chiropractic - December 13th, 2009 [December 13th, 2009]
- Cell phones and cancer again, or: Oh, no! My cell phone’s going to give me cancer! (revisited) - December 20th, 2009 [December 20th, 2009]
- Another wrinkle to the USPSTF mammogram guidelines kerfuffle: What about African-American women? - December 20th, 2009 [December 20th, 2009]
- Acupuncture, the P-Value Fallacy, and Honesty - December 20th, 2009 [December 20th, 2009]
- The One True Cause of All Disease - December 20th, 2009 [December 20th, 2009]
- Communicating with the Locked-In - December 20th, 2009 [December 20th, 2009]
- Are the benefits of breastfeeding oversold? - December 20th, 2009 [December 20th, 2009]
- Measles - December 20th, 2009 [December 20th, 2009]
- Radiation from medical imaging and cancer risk - December 21st, 2009 [December 21st, 2009]
- Multiple Sclerosis and Irrational Exuberance - December 21st, 2009 [December 21st, 2009]
- Medical Fun with Christmas Carols - December 22nd, 2009 [December 22nd, 2009]
- Lithium for ALS – Angioplasty for MS - December 23rd, 2009 [December 23rd, 2009]
- “Toxins”: the new evil humours - December 24th, 2009 [December 24th, 2009]
- 2009’s Top 5 Threats To Science In Medicine - December 24th, 2009 [December 24th, 2009]
- Buteyko Breathing Technique – Nothing to Hyperventilate About - December 26th, 2009 [December 26th, 2009]
- The Graston Technique – Inducing Microtrauma with Instruments - December 29th, 2009 [December 29th, 2009]
- The “pharma shill” gambit - December 29th, 2009 [December 29th, 2009]
- Ginkgo biloba – No Effect - December 30th, 2009 [December 30th, 2009]
- Oppose “Big Floss”; practice alternative dentistry - January 1st, 2010 [January 1st, 2010]
- Causation and Hill’s Criteria - January 3rd, 2010 [January 3rd, 2010]
- The life cycle of translational research - January 10th, 2010 [January 10th, 2010]
- The anti-vaccine movement strikes back against Dr. Paul Offit - January 10th, 2010 [January 10th, 2010]
- Osteoporosis Drugs: Good Medicine or Big Pharma Scam? - January 10th, 2010 [January 10th, 2010]
- Acupuncture for Hot Flashes - January 10th, 2010 [January 10th, 2010]
- The case for neonatal circumcision - January 10th, 2010 [January 10th, 2010]
- A victory for science-based medicine - January 10th, 2010 [January 10th, 2010]
- James Ray and testosterone replacement therapy (TRT) - January 10th, 2010 [January 10th, 2010]
- The Water Cure: Another Example of Self Deception and the “Lone Genius” - January 12th, 2010 [January 12th, 2010]
- Be careful what you wish for, Dr. Dossey, you just might get it - January 13th, 2010 [January 13th, 2010]
- You. You. Who are you calling a You You? - January 15th, 2010 [January 15th, 2010]
- The War on Salt - January 16th, 2010 [January 16th, 2010]
- Is breech vaginal delivery safe? - January 16th, 2010 [January 16th, 2010]