An article written by 3 chiropractors and a PhD in physical education and published on December 2, 2009 in the journal Chiropractic and Osteopathy may have sounded the death knell for chiropractic.

The chiropractic subluxation is the essential basis of chiropractic theory. A true subluxation is a partial dislocation: chiropractors originally believed bones were actually out of place. When x-rays proved this was not true, they were forced to re-define the chiropractic subluxation as “a complex of functional and/or structural and/or pathological articular changes that compromise neural integrity and may influence organ system function and general health.” Yet most chiropractors are still telling patients their spine is out of alignment and they are going to fix it. Early chiropractors believed that 100% of disease was caused by subluxation. Today most chiropractors still claim that subluxations cause interference with the nervous system, leading to suboptimal health and causing disease.

What’s the evidence? In the 114 years since chiropractic began, the existence of chiropractic subluxations has never been objectively demonstrated. They have never been shown to cause interference with the nervous system. They have never been shown to cause disease. Critics of chiropractic have been pointing this out for decades, but now chiropractors themselves have come to the same conclusion.

In “An epidemiological examination of the subluxation construct using Hill’s criteria of causation” Timothy A. Mirtz, Lon Morgan, Lawrence H. Wyatt, and Leon Greene analyze the peer-reviewed chiropractic literature in the light of Hill’s criteria, the most commonly used model for evaluating whether a suspected cause is a real cause. They ask whether the evidence shows that chiropractic subluxations cause interference with the nervous system and whether they cause disease. The evidence fails to fulfill even a single one of Hill’s nine criteria of causation. They conclude:

There is a significant lack of evidence in the literature to fulfill Hill’s criteria of causation as regards chiropractic subluxation. No supportive evidence is found for the chiropractic subluxation being associated with any disease process or of creating suboptimal health conditions requiring intervention. Regardless of popular appeal this leaves the subluxation construct in the realm of unsupported speculation. This lack of supportive evidence suggests the subluxation construct has no valid clinical applicability. [emphasis added]

While some chiropractors have rejected the subluxation paradigm, it is supported by the major chiropractic organizations and schools and is considered essential by the great majority of practicing chiropractors. In two recent studies cited in the Mirtz et al. article, 98% of chiropractors believed that “most” or “many” diseases were caused by spinal misalignments and over 75% of chiropractors believed that subluxation was a significant contributing factor to 50% or more of visceral disorders (such as asthma and colic), an implausible idea that is not supported by any evidence whatsoever. Simon Singh was sued for saying so when he correctly referred to “wacky ideas” and “bogus treatments.”

When chiropractors use spinal manipulation therapy for symptomatic relief of mechanical low back pain, they are employing an evidence-based method also used by physical therapists, doctors of osteopathy, and others. When they do “chiropractic adjustments” to correct a “subluxation” for other conditions, especially for non-musculoskeletal conditions or “health maintenance,” they are employing a non-scientific belief system that is no longer viable.

As the authors of this paper indicate, the subluxation construct must go. And without the subluxation, the whole rationale for chiropractic collapses, leaving chiropractors no justifiable place in modern medical care except as competitors of physical therapists in providing treatment of certain musculoskeletal conditions.

The absence of publicity is astounding. This study has not even been noticed by the media. Where are the sensationalist journalists who usually exaggerate the news and make up provocative headlines? They could be trumpeting “Chiropractic Is Dead!” “Chiropractors Admit They Were Deluded by False Beliefs” “Simon Singh Vindicated: Chiropractic Really Is Bogus” and so on. Chiropractors demolishing the basis for chiropractic ought to be big news.

When the news finally gets out, I predict contorted efforts at damage control. Chiropractors will claim that it is not appropriate to apply the Hill criteria in this way, and that the criteria are not a valid test of causality. That’s a straw man: not even Hill suggested that the criteria were a definitive test. They are more of a guide to thinking about causality. Edzard Ernst, the world’s first professor of complementary and alternative medicine, finds them useful. He has recently applied Hill’s criteria to neck manipulation as a cause of stroke: he found that it fulfilled all but one of the criteria for causation. (Article pending publication). Chiropractors won’t like that either.

I predict the authors of this paper will be attacked as traitors by their colleagues. And I predict my own comments will be misinterpreted as some kind of personal vendetta and I will be called ugly names. I also predict that no one will dispassionately offer acceptable scientific evidence to contradict the findings of the paper (They can’t, because there isn’t any!). The first comment (and so far the only comment) on the Chiropractic and Osteopathy website offers no counter-evidence but rather calls for not letting evidence-based protocols overshadow clinical experience.(!) The Weekly Waluation of the Weasel Words of Woo could have a lot of fun translating that statement.

If chiropractors reject the conclusions of the Mirtz et al. paper, the burden of proof falls on them to show

1.  that the subluxation can be objectively demonstrated,
2.  that it does cause interference with the nervous system, and
3.  that it does cause disease.

They have failed to do so for 114 years.

Most chiropractic research falls under the category of Tooth Fairy Science. Instead of doing good basic research to examine the subluxation construct as a falsifiable hypothesis, they blindly forged ahead, implemented it for diagnosis and treatment, and studied various aspects of its clinical use.

The chiropractic emperor has no clothes, and now even some chiropractors have realized that. This study should mark the beginning of the end for chiropractic, but it won’t. Superstition never dies, particularly when it is essential to livelihood.

We are nearing the end of the second wave of the 2009 H1N1 pandemic, and are now a few months out from the release of the vaccine directed against it.  Two topics have dominated the conversation: the safety of the 2009 H1N1 influenza vaccine, and the actual severity of the 2009 H1N1 infection.  Considering the amount of attention SBM has paid the pandemic and its surrounding issues, and in light of a couple of studies just released, it seems time for an update.

2009 H1N1 Vaccine Safety

This week the CDC released a report that evaluated the safety record of the 2009 H1N1 vaccine.  The first two months of the vaccine’s use were examined, from October 1^st through November 24^th using data from two of the larger surveillance systems monitoring the 2009 H1N1 vaccine’s safety: the Vaccine Adverse Event Reporting System (VAERS) and the Vaccine Safety Datalink (VSD).  This report represents the largest, and to date best, evaluation of the 2009 H1N1 vaccine’s safety profile since its initial testing and release.  The findings are reassuring.

We’ve talked about VAERS’ uses (and abuses) in the past.  Nevertheless, used properly as a surveillance tool, a “canary in a coal mine,” it can be quite helpful.  In that two-month span of time when 46.2 million doses of H1N1 vaccine were distributed, 3,783 adverse events associated with it were reported to VAERS.  204 of these events were classified as “serious,” including 13 deaths that occurred within 19 days of vaccine administration.

At first blush people may assume (unwisely) that the vaccine directly caused each of these reported events, and would thus yield an adverse event rate of 82 total adverse events and 4.4 serious adverse events per 1 million doses.  This is indeed the assumption (and mistake) made by people claiming for instance that the flu vaccine has caused X number of deaths or Y cases of Guillain Barre Syndrome (GBS).  Even taken (again, unwisely) at face value, these rates would be impressively low, particularly when compared to the risks of H1N1 infection, as we shall see later.

The story is even more reassuring once we look properly at the data.  It bears repeating that VAERS does not (nor was it meant to) establish causation, it only holds the potential to suggest a correlation.  We should also bear in mind that GBS, death, all adverse events in fact, occur at a baseline rate in the population in the absence of the vaccine (a hypothetical vaccine causing zero adverse events would still have a list of adverse events reported to VAERS reflecting the population’s baseline rates).  Thus to even determine if there is a significant correlation between the vaccine and any given adverse event, we need to determine not only how many adverse events occur in relation to the 2009 H1N1 vaccine, but the number that occur above the expected baseline.

That having been said, let’s examine the most concerning number first, the 13 reported deaths.  Each of the 13 are detailed on this chart.  It’s very much worth taking a look.  There is no discernable pattern to the ages of these unfortunate people, their underlying diseases, or their causes of death.  9 of these 13 people had significant underlying diseases, and one of them died in a car accident. Indeed, considering the population’s baseline mortality rate, it’s remarkable that only 13 people out of 46.2 million died within 3 weeks of receiving the vaccine by chance alone.  This doesn’t definitively exonerate the 2009 H1N1 vaccine from these deaths (well, we can probably safely rule out the car accident), but it certainly makes its involvement highly unlikely.

H1N1 Vaccine and GBS?

What of the concern of Guillain Barre Syndrome (GBS) following vaccine administration?  After all, at least one influenza vaccine in the last three decades has been shown to cause GBS in rare cases, and some poorly handled stories in the media have further elevated public concern.

The first two months of vaccine use saw 12 cases of suspected GBS reported to VAERS.  Investigation into these reports has confirmed four of these to be cases of GBS, four were not GBS, and the final four are still under scrutiny.

Again, these cases require context.  As the baseline rate of GBS is ~1/100,000 people per year, ~550 cases can be expected to occur in the US during the two months of this report.  These 8 likely cases of GBS in 46.2 million doses of vaccine is certainly not higher (and is in fact far less) than what one would expect to see by chance.  The VAERS database provides no reason to suspect the 2009 H1N1 vaccine has anything but chance correlation with cases of GBS.

H1N1 and Other Severe Adverse Events?

There is no correlation between the H1N1 vaccine and either GBS or death, but what of other concerning adverse events?  An evaluation of the 204 serious events reported reveals a scattershot of diseases, none of which have a signal that rises above baseline rates.

The CDC report contains a similar analysis using data from the VSD, where 438,376 doses of the H1N1 vaccine had been administered and adverse events tracked.  As with the VAERS data, no serious adverse events rose above their baseline rates.

In short, after the first two months of use and 46.2 million doses, the VAERS and VSD data fails to provide any evidence to correlate the 2009 H1N1 vaccine to any serious adverse event.  Given the seasonal influenza vaccine’s similar record over the past several decades, that the 2009 H1N1 vaccine continues to display an exemplary safety profile is not unexpected, but it is reassuring.

How Severe is 2009 H1N1?

What of H1N1’s severity?  What toll has it exacted?  The CDC has made detailed information, updated weekly, available to the public on its Fluview website.  Containing a wealth of information, there you can see 2009 H1N1’s unique and peculiar epidemiology, the unseasonable spikes in outpatient visits for influenza-like illnesses that have troubled our EDs for the last few months, and the trend of lab-confirmed influenza hospitalizations and mortality over time.

Hard numbers are also available.  As of November 28^th, at least 31,320 people in the US have been hospitalized and 1,336 have died from 2009 H1N1 since August 30^th.  The 2009 H1N1 has thus far claimed the lives of at least 250 children in between the traditional flu seasons, which is more than the two prior flu seasons combined.

This data is most helpful if viewed as the minimum confirmed impact of the disease, and as a catalogue of the most severe cases to date.  What you will not find on the Fluview site is the actual incidence of influenza infection, the total number of people infected, including minor infections.  This number is extremely valuable when trying to gauge the true severity of any infection, but fiendishly difficult to acquire.

A study published in PLoS Medicine this week contains one of the latest attempts to quantify 2009 H1N1’s severity to date.  Drawing from the data of two US cities during the initial wave of infections between April and July, they estimated that of all 2009 H1N1 infections, between 0.16-1.44% will require hospitalization, 0.028-0.239% will require ICU care, and 0.007-0.048% will die.

This study has garnered a significant amount of attention, for its estimates of severity are considerably lower (thankfully) than those made by the President’s Council of Advisors on Science and Technology in early August.  The accuracy and differences between these estimates, the inherent difficulty of determining the true incidence, severity, and future course of diseases like influenza warrants its own post, and I’ll not address this particular angle in greater depth here.

I’d like to instead reflect on what these two studies might tell us about the risks of contracting 2009 H1N1 compared to the risks of receiving the vaccine to protect against it.

On the one hand, we have a virus that has proven itself to be widespread and highly contagious, to have claimed the lives of at least 1,336 and hospitalized over 30,000.  Conservative estimates from the PLoS study place one’s risk of hospitalization if infected at ~1/625, and risk of death ~1/14,285.  Furthermore, though we have completed the second wave of the pandemic, a third wave is almost certain to come.  A small minority of the population has thus far been infected, past influenza pandemics have featured a triple peak, and we have now entered the beginning of the traditional influenza season.

On the other hand, we have an inexpensive vaccine which is an excellent match to this strain, generates an appropriate antibody response in most people (particularly those in the highest risk groups for 2009 H1N1), and after over 46 million doses has yet to be significantly correlated with any severe adverse events.

Conclusion

There are still a lot of uncertainties regarding the rest of this influenza season.  Will we have a third peak of H1N1, and if so, how severe will it be?  Will it continue to preferentially afflict the young, or will the elderly suffer a greater impact than they have to date?  How will the presence of 2009 H1N1 impact the normal flu season, will it be cumulative, or will 2009 H1N1 “crowd out” the seasonal strains?  The list goes on, and it absolutely includes the possibility that with ongoing surveillance and studies we may identify a serious but rare side effect caused by the vaccine.

As time goes on we will continue to refine our knowledge of influenza, and these questions will be answered, but it is unlikely that the big picture will significantly change.  Influenza is a virus with serious potential for harm that can be prevented by one of the safest interventions in modern medicine.  Please, particularly if you or yours are in a high-risk group, get vaccinated; I already know far too many of the names on this list.

Steve Novella whimsically opined on a recent phone call that irrationality must convey a survival advantage for humans. I’m afraid he has a point.

It’s much easier to scare people than to reassure them, and we have a difficult time with objectivity in the face of a good story. In fact, our brains seem to be hard wired for bias – and we’re great at drawing subtle inferences from interactions, and making our observations fit preconceived notions. A few of us try to fight that urge, and we call ourselves scientists.

Given this context of human frailty, it’s rather unsurprising that the recent USPSTF mammogram guidelines resulted in a national media meltdown of epic proportions. Just for fun, and because David Gorski nudged me towards this topic, I’m going to review some of the key reasons why the drama was both predictable and preventable.  (And for an excellent, and more detailed review of the science behind the kerfuffle, David’s recent SBM article is required reading.)

Preamble

In an effort to increase early detection of breast cancer, American women have been encouraged to get annual screening mammograms starting at age 40. Even though mammograms aren’t as sensitive and specific as we’d like, they’re the best screening test we have – and so with all the caveats and vagaries associated with what I’d call a “messy test,” we somehow collectively agreed that it was worth it to do them in this age group.

Now, given the life-threatening nature of breast cancer, it’s only natural that advocacy groups and professional societies want to do everything in their power to save women from it. So of course they threw all their weight behind improving compliance with screening mammograms, and spent millions on educating women about the importance of the test. Because, after all, there is no good alternative.

However, the downside of an imprecise test is the false positive results that require (in some cases) invasive studies to refute them.  And so this leaves us with 2 value judgments:

1. How many women is it acceptable to harm (with unnecessary biopsies) in order to save one life? Roughly, the answer is a maximum of 250 over 10 years (I came up with that number from the data here -  if as many as half of women receive a “false alarm” mammogram over a period of 10 years of testing, and half of those undergo an unnecessary biopsy).

2. How many tests are we willing to do (this is more-or-less an economic question) to save 1 life?  The answer is roughly 1900.

So when the USPSTF took a fresh look at the risks and benefits of mammography and recommended against screening average risk women between 40-50 (and reducing mammogram frequency to every other year for those over 50), what they were saying is that they would rather lose a few lives to save a vast number of injuries (unnecessary biopsies) and costly annual testing. In fact, their answer was that they were willing to perform 1300 mammograms to save 1 life, not 1900 (as has been our standard of care).

This value judgment is actually not, in and of itself, earth shattering or irresponsible. But it’s the societal context into which this judgment was released that made all the difference.

Timing Is Everything -Or – Why Not To Bring A Party Hat To A Funeral

First of all, the timing of the USPSTF guidelines couldn’t have been worse. The country was in the midst of trying to pass our first serious healthcare reform bill in decades (at least, the house reform bill was being voted upon the week that the USPSTF guidelines were released) and opponents of the bill had already expressed vehement concern about arbitrary government rationing of healthcare services.

Is there a more inopportune moment for a government agency to (against the commonly held views of the rest of the medical establishment) recommend reduction in frequency of a life-saving screening test for women? The fact that the vice chair of the USPSTF (Dr. Diana Petitti) said she hadn’t thought about the greater context when she scheduled the press release is quite astonishing. On the one hand, I suppose it shows that the workgroup wasn’t particularly politically biased. On the other hand, it violates Public Relations 101 so completely as to call into question the judgment of those making… er… judgments.

You Can’t Replace Something With Nothing -Or – How To Take Scissors From A Baby

Let’s just say for a moment that we all agree that mammograms aren’t the greatest screening test for breast cancer. They’re rather expensive, and wasteful. Perhaps one might even argue that in a healthcare system with limited resources, one healthy woman’s screening test is another woman’s insulin.  But – it’s all we have. And they do save lives… occasionally.

Anyone who’s seen a child pick up something harmful realizes that the only way to take it from them without tears is to replace it with something harmless. You can’t just take away mammograms from women who have come to expect it, without offering them something more sensible. If there is nothing, then I’m afraid that discontinuing them will result in considerable outrage which you may or may not wish to engage. Given the size and power of the breast cancer lobby – I’d say it’s pretty much political suicide. (And of course, after the USPSTF released their guidelines, HHS Secretary Kathleen Sebelius virtually denounced it, and congress moved immediately to create legislation to require health insurers to cover mammograms for women in their 40s-50s).

Know Your Opposition -Or- Don’t Bring A Knife To A Gun Fight

And that brings me to my next point. The breast cancer movement is one of the most powerful and successful disease fighting machines in the history of medicine. And bravo to all the women and men who made it such a visible disease. The amount of funding, research, and PR that this cancer gets is astounding – it dwarfs many other worthy diseases (like pancreatic cancer or lymphoma), and is a force to be reckoned with.

Which is why, before you undermine a cherished tenet of such a group, you take a long hard look at what you’re going to say… Because it will be shouted from the hilltops, scrutinized from every conceivable angle, and used to rally all of Hollywood, the medical establishment, and everyone in Washington to its cause. Yeah, you better be darn sure you’re “right” (whatever that means in this context) before attempting to promote a service cut back to this group.

Know Who You Are -Or- Unilateral Decision Making Is Not A Great Idea, Especially For Government

And finally, it’s important not only to know who you’re dealing with, but to know your mission in society so you can be maximally effective. The US government exists to honor the will of the people and serve its citizens. The best way to do that is to listen to them carefully, engage in consensus-building, and try to be a good steward of resources. When government behaves in ways counter to our expectations, it provokes legitimate negativity.

So, for example, when a small group of civil servants hole themselves up in a room to create guidelines that will potentially take preventive health services away from women – resulting in a larger number of deaths each year, they don’t invite input from key stakeholders, and then announce their views in the midst of a firestorm about “rationing,” you’re going to get nuclear level blowback.

Summary

The new USPSTF guidelines for mammogram screenings debacle serves as a perfect public relations case study in what not to do in advancing healthcare reform. It was the perfect storm of high profile subject, bad timing, poor argument preparation, and lack of back up planning. Though we could have had a rational discussion about the cost/benefit analysis of this particular screening test, what we got instead was the appearance of a unilateral rationing decision by an out-of-touch government organization, devaluing women to the point of death. Throw that chum in the water of human frailty and you’ll get the same result every time: a media feeding frenzy that makes you regret the moment that guideline development became a twinkle in your task-force eye.

The new mantra of midwives and their advocates is “evidence based practice.” Lamaze, the childbirth education organization has changed the name of their blog to “Science and Sensibility” emphasizing the importance of science and promising:

Lamaze education and practices are based on the best, most current medical evidence available, and can help reduce the overuse of unnecessary interventions while improving overall outcomes for mothers and babies.

But midwives and childbirth educators like Lamaze have a problem. The scientific evidence often conflicts with their ideology. They could address this problem in several ways. Midwives could modify their specific ideological beliefs on the basis of scientific evidence. Childbirth educators could question whether ideology has had an inappropriate impact on the promulgation and validation of their recommendations. Both those approaches would involve a threat to cherished beliefs. They, therefore, have taken a different approach. They’ve tried to justify ignoring scientific evidence.

As midwives Jane Munro and Helen Spiby have documented in The Nature and Use of Evidence in Midwifery, the first chapter of their book Evidenced Based Midwifery, midwives were initially enthusiastic about basing clinical practice on scientific evidence. That’s because they had long told each other that midwifery was “science based” while obstetrics was not:

At the beginning of the evidence based practice movement, much of the midwifery profession responded enthusiastically to the potential for change… Evidence based practice was seen to be offering a powerful tool to question and examine obstetric-led models of care that had dominated the previous decades. The results of such examination could have meant ’starting stopping’ the unhelpful interventions that had embedded themselves in common practice …

But it turned out that obstetrics had been based on scientific evidence all along and it was midwifery that ignored the scientific evidence in favor of ideology. Indeed, almost all practices exclusive to midwifery (as opposed to copied from obstetrics) have never been tested. They might be valuable; they might be useless; they might even be harmful. No one bothered to check before implementing them because they were based on ideology.

It has been quite a shock to midwives and childbirth educators to learn that most of their own practices have never been scientifically validated. Even worse, much of their critique of modern obstetrics flies in the face of the existing scientific evidence. As Munro and Spiby explain:

… [S]ome midwives have not been so enthusiastic [about evidence based practice], viewing the drive to create and implement evidence as a threat to their clinical freedom.

In other words, cherished ideological beliefs conflict with scientific evidence. Thus began the attack on scientific evidence.

As a first approach, midwives and childbirth educators have rejected the definition of evidence. As defined by Sackett, the founder of evidence based practice, it is “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.” That sounds objective, and evidently, objectivity is a problem. They have attempted to solve that problem by insisting that evidence can only be defined in context. “Context” in this case really means “ideology.”

Scientists see the ideology free nature of scientific evidence as one of its strengths and therefore privilege it as the ideal form of evidence. But Lomas, writing about midwifery critics of evidence, explains that they reject this privileged status:

[I]t is important that context evidence should not be viewed as any less ’scientific’. They advocate moving forward from the epistemological argument about what is ‘best evidence’ towards a ‘balanced consensus’ …

The use of the word “consensus” is illuminating. Evidence can only be evidence if it includes the opinions of midwives and childbirth educators, whether those opinions are based on science or not. Indeed, the scientific facts are merely one aspect of evidence. “Social science oriented research” and “the views of stakeholders” are supposed to have equivalent weight.

Such is the genesis of midwifery papers like Wickham’s Evidence Informed Midwifery, and, my personal favorite, Parrat and Fahy’s Including the nonrational is sensible midwifery. When the evidence does not support your claims, the use of adjuncts, including nonrational ones, will justify any beliefs.

The Parrat and Fahy paper is particularly instructive on this point. Their central claim is that the inclusion of the non-rational is midwifery “enhances safety”:

When the concept of ’safety’ is considered in childbearing it can illustrate how insensible rationality can be and how negative consequences can occur. Safety is an abstract concept because it is difficult to define and can only be considered in general terms. Rational dichotomous thought, however, provides ’safety’ with the following defining boundaries:

- ’safe’ has a precise opposite called ‘unsafe’,
- every situation/person/thing must be either be safe or unsafe,
- a situation/person/thing cannot be both safe and unsafe,and
- it is not possible for a situation/person/thing to be anything
other than safe or unsafe.

Furthermore:

…What is deemed as safe is aligned with what is rational and what is unsafe is aligned with what is irrational. As irrationality is not acceptable this essentially forces the definition of safety to be thought of as ‘true’ even though it may not fit with personal experience and all situations… As the standard birth environment is the medicotechnical environment of the hospital this is presumed to be the safest. Its ‘opposite’, the home environment, is therefore rationalised to be unsafe. To argue otherwise would define the rational person as irrational… In the purely rationalist way of thinking there is no other option except to consider that honouring the nonrational variabilities of individual bodily experience is irrational and unsafe.

The authors end with a flourish of outright stupidity:

For example, when a woman and midwife have agreed to use expectant management of third stage, but bleeding begins unexpectedly, the expert midwife will respond with either or both rational and nonrational ways of thinking. Depending upon all the particularities of the situation the midwife may focus on supporting love between the woman and her baby; she may call the woman back to her body; and/or she may change to active management of third stage. It is sensible practice to respond to in-the-moment clinical situations in this way…

This paper is but one example of a disturbing trend: many midwives and childbirth educators use the term “scientific evidence” merely as a rhetorical device, in the same way that creationism and other form of pseudoscience use the term “scientific evidence.” As Coker details in his article Distinguishing Science and Pseudoscience:

Pseudoscience appeals to the truth-criteria of scientific methodology while simultaneously denying their validity.

Similarly, midwives and childbirth educators invoke the criteria of scientific methodology while simultaneously insisting that their personal beliefs matter as much if not more.

Several weeks ago I wrote the first in a brief series of posts discussing the different types of evidence used in medicine. In that post I discussed the role of correlation in determining cause and effect.

In this post I will discuss the basic features of an experimental study, which can sere as a check-list in evaluating the quality of a clinical trial.

Medical studies can be divided into two main categories – pre-clinical or basic science studies, and clinical studies. Basic science studies involve looking at how parts of the biological system work and how they can be manipulated. They typically involve so-called in vitro studies (literally in glass) – using test tubes, petri dishes, genetic sequencers, etc. Or they can involve animal studies.

Clinical trials involve people. They are further divided into two main categories – observational studies and experimental studies. I will be discussing experimental studies in this post – studies in which an intervention is done to study subjects. Observational studies, on the other hand, look at what is happening or what has happened in the world, but does not involve any intervention.

Experimental Studies

The primary advantage of experimental studies is that they allow for the direct control of variables – in the hopes of isolating the variable of interest. Results are therefore capable of being highly reliable, although good clinical experiments are difficult to design and execute. When assessing a clinical trial here are the features to examine.

Prospective vs Retrospective

A prospective trial is one in which the treatment and the outcomes are determined prior to any intervention being done. Experimental trials are almost by definition prospective. A retrospective trial is one in which the data is gathered after the fact – taking patient records, for example, and looking at treatments and outcomes.

In a retrospective study you can try to account for variables, but you cannot control for them. It is therefore much more likely that there are confounding factors and the results are not as reliable.Also, retrospective studies can be biased by the way information is obtained – there can be a bias in the way patients are identified, for example.

Prospective trials are therefore considered superior to retrospective trials, which are at best preliminary in their conclusions.

Placebo-Controlled

Not all prospective trials are placebo-controlled, however. A non-controlled trial might identify potential subjects, give them all a treatment, and then see how they do. Such open-label single arm trials cannot control for placebo effects or experimenter biases, and again results should be considered preliminary.

Open or uncontrolled trials are not useless, however. The outcome of subjects in such trials can be compared to historical controls, and if a significant result is apparent (along with safety) can be used to justify a larger and more rigorous trial.

Controlled trials have one or more comparison groups in the trial itself – different groups of subjects receive different treatments or no treatment. All subjects can be followed in same manner. Control groups allow the experimenter to make sure that all the subjects have the same disease or symptoms, that they receive known treatments, and many variables (such as other treatments they may be receiving, severity at inclusion, age, sex, race, etc.) can be accounted for.

Controlling for variables

With controlled trials the experimenter can start to control for variables. If the question is – does treatment A improve outcome in disease X, a controlled prospective trial can attempt to isolate treatment A from other factors that may affect outcome.

One method for controlling variables is stratification – the study protocol can place subjects in different treatment groups so that the groups end up with the same proportion of different sexes, ages, races, and other known variables that may be pertinent. Stratification can control for known or obvious confounding factors.

But of course there can always be unknown confounding factors. The only way to deal with these is through randomization and large study size. If a large number of subjects are randomly assigned (once stratified for age, sex, etc.) into the different treatment groups, then any unknown variables should average out. Of course, this requires sufficient numbers – small studies are always suspect because the groups may be significantly different by chance alone.

Randomization is important because when patients select their own treatments this opens the door for selection bias. For example, sicker patients may opt for more aggressive therapy. They will do worse because they were sicker to begin with, making the more aggressive therapy look less effective.

Blinding

A randomized prospective trial can control for many variables, but the only way to control for placebo effects and the bias of the experimenters is with blinding – meaning that participants don’t know who is getting the real treatment and who is getting a different treatment or a placebo.

A single-blind study is one in which subjects do not know which treatment they are getting. A double-blind study is one in which the experimenter does not know either – until the study is done and the “code is broken.”

When subjects are blinded, placebo effects should be the same. It is often difficult, however, to fully blind subjects. Medications may have obvious side effects, and subjects who experience the side effects know they are getting active medication.

Physical interventions, like acupuncture, surgery, massage, or physical therapy, are difficult to impossible to blind. A person knows if they have been massaged or not. For these studies creative blinding techniques may need to be used. Or, “sham” procedures can be used for placebos.

Studies may also assess how successful the blinding was – by asking subject if they think they received the placebo or the treatment.

Experimenters also need to be blinded to eliminate placebo and biasing effects. This is easy for drug trials, but may be impossible for physical intervention trials. However, a study can be partially double-blinded if there is a blinded evaluator – an experimenter whose only involvement with the study is to assess the subjects, while carefully avoiding any information that would clue them in as to which treatment arm each subject was in.

But the best studies are ones in which everyone involved is completely blinded until the results are completely in.

Outcome measures

Deciding how to determine if an intervention “works” is not always trivial. Outcome measure need to be a good and reliable marker of the disease or syndrome you are following. For example, in a diabetes study, do you follow HgA1C, random glucose checks, glucose tolerance tests, end-organ damage, need for medication, or some other biological marker?

In addition to being a good marker for what you are studying, the outcome should be meaningful. Do we care if a cholesterol lowering drug lowers total cholesterol, or if it prevents heart attacks and strokes? And if it prevents heart events, does it prolong survival (or just reduces angina without affecting survival)?

Outcomes also need to be free of confounding. For example, early stroke trials looked at stroke incidence, which may seem reasonable. However, if more subjects on a treatment died of heart attacks, they would not be around to have a stroke, so the treatment reduces stroke but only by allowing more heart attacks. So stroke-free survival is a better outcome to follow.

Outcome measures also vary on how objective or subjective they are. Just asking patients how they feel is not a very reliable outcome measure. You can pseudo-quantify this by asking them to put a number on their pain or other symptoms, but it is still a subjective reports. Measuring the volume of lesions in the brain, however, is an objective outcome measure, and is therefore more reliable.

Many studies will follow several outcomes – some subjective but important, and others objective and quantifiable if an indirect marker rather than a direct outcome we care about.

Statistical analysis

I won’t go into statistics in any detail, as that is a highly technical area and any reasonable treatment would be much longer than the rest of this post. Here even medical professionals rely upon statistical experts to make sure we get it right.

But it is good to understand the basics (as long as you don’t rely upon basic knowledge – then it is easy to be fooled by fancy statistical tricks).

The most basic concept of clinical trials is statistical significance – is there an effect or correlation that is probably greater than chance. Most studies rely upon the P-value, which is a measure of the chance the result occurring if the the null hypothesis (no effect) is correct. A P-value of 0.05 means (roughly) that 5% (or 1 in 20) probability that the outcome is due to chance alone, and not a real effect. P-value of 0.05 is commonly used as a cutoff of statistical significance, but it is important to realize with this cutoff 1 in 20 studies of worthless treatments will appear positive due to chance alone. Lower P-values, such as 0.01, are more significant.

But P-value isn’t everything.  A poorly designed study can result in an impressive P-value. Also, the size of the effect must be considered. You can have a low P-value for a tiny effect (if there are large numbers of subjects in the trial) – the effect may be clinically insignifcant, and small effects are more likely to be due to hidden biases or confounders.

Therefore, we generally are only impressed when a clinically large effect also has a low P-value.

In addition to P-value, the number of subjects in the trial is very important. Even though these are related, the larger the study the more impressive the results, as random fluctuations are less likely to play a role.

One common trick to look out for is multiple analysis.  A study may, for example, look at 10 variables (or one variable at 10 different points in time), and find statistical significance for one, and present that as a positive study. However, this is equivalent to taking 10 chances at that 1 in 20 chance of hitting significance. Proper statistical analysis will account for multiple comparisons.

Other factors to look out for

There are features that are important to consider is evaluating a clinical trial. What was the dropout rate? If half of the subjects dropped out, that unrandomizes or biases the groups, because drop outs are not random. For example, subjects that do not respond to treatment may drop out, leaving only those who do well.

Not all controls are equal as well. Sometime the control group is not an inactive placebo but standard care. What if the standard treatment is too effective, or what if it is not effective at all. You need to know what the study treatment is being compared to.

Conclusion

When a new clinical trial is being promoted in the news as evidence for or against a treatment – run down this list. Is it a randomized, controlled, double-blind trial, is the blinding adequate, are the outcome measures objective and relevant, is the effect size robust, how large is the study, what variables are actually being isolated, and what was the drop out rate?

And of course, no one study is ever the definitive last word on a clinical question. Each study must be put in the context of the full scientific literature, which means considering plausibility or prior probability. That is the essence of science-based medicine.