A new study published in PLOS Biology looks at the potential magnitude and effect of publication bias in animal trials. Essentially, the authors conclude that there is a significant file-drawer effect – failure to publish negative studies – with animal studies and this impacts the translation of animal research to human clinical trials.

SBM is greatly concerned with the technology of medical science. On one level, the methods of  individual studies need to be closely analyzed for rigor and bias. But we also go to great pains to dispel the myth that individual studies can tell us much about the practice of medicine.

Reliable conclusions come from interpreting the literature as a whole, and not just individual studies. Further, the whole of the literature is greater than the sum of individual studies – there are patterns and effects in the literature itself that need to be considered.

One big effect is the file-drawer effect, or publication bias – the tendency to publish positive studies more than negative studies. A study showing that a treatment works or has potential is often seen as doing more for the reputation of a journal and the careers of the scientists than negative studies. So studies with no measurable effect tend to languish unpublished.

Individual studies looking at an ineffective treatment (if we assume perfect research methodology) should vary around no net effect.  If those studies that are positive at random are more likely to be published than those studies that are neutral or negative, than any systematic review of the published literature is likely to find a falsely positive effect.

Of course, we do not live in a perfect world and many studies have imperfect methods and even hidden biases. So in reality there is likely to be a positive bias to the studies.  This positive bias magnifies the positive publication bias.

There are attempts in the works to mitigate the problem of publication bias in the clinical literature. For example, clinicaltrials.gov is a registry of all trials involving human subjects – before the trials are completed and the results known. This way reviewers can have access to all the data – not just the data researchers and journal editors deem worthy.

This new study seeks to explore if publication bias is similarly a problem with animal studies. The issues are similar to human trials. There is an ethical question, as sacrificing animals in research is justified by the data we get in return. If that data is hidden and does not become part of the published record, than the animals were sacrificed for nothing.

And also, publication bias can lead to false conclusions. This in turn can, for example, lead to clinical trials of a drug that seems promising in animal studies. This could potentially expose human subjects to a harmful or just worthless drug that would not have made it to human trials if all the negative animal data were published.

The study itself looked at a database of animal models of stroke. They examined 525 publications involving 16 different stroke interventions. There are a few different types of statistical analysis that can be done to infer probable publication bias. Basically, without publication bias there should be a certain distribution of findings in terms of effect sizes. If only positive or larger effect sizes are being published, then the distribution will be skewed.

This type of analysis provides an estimation only. They found that:

Egger regression and trim-and-fill analysis suggested that publication bias was highly prevalent (present in the literature for 16 and ten interventions, respectively) in animal studies modelling stroke. Trim-and-fill analysis suggested that publication bias might account for around one-third of the efficacy reported in systematic reviews, with reported efficacy falling from 31.3% to 23.8% after adjustment for publication bias. We estimate that a further 214 experiments (in addition to the 1,359 identified through rigorous systematic review; non publication rate 14%) have been conducted but not reported. It is probable that publication bias has an important impact in other animal disease models, and more broadly in the life sciences.

So there was some disagreement between the methods used, but both showed that there is likely to be a significant publication bias. If their analysis is correct, about one third of systematic reviews of animal studies in stroke that conclude an intervention works may be due to publication bias rather than a real effect. The authors also speculate that this effect is likely not unique to stroke, and may be generalizable to animal studies in general.

Of course, this is just an individual study, and further analysis using different data sets are needed to confirm these results.

Conclusion

The results of this study are not surprising and are in line with what is known from examining clinical trials. They suggest that similar methods to minimize publication bias are necessary for animal studies in addition to human trials.

Hopefully, this kind of self-critical analysis will lead to improvement in the technology of medical research. It should further lead to more caution in interpreting not only single studies but systematic reviews.

Also, in my opinion, it highlights the need to consider basic science and plausibility in evaluating animal and clinical trials.

A panels of bloggers from SBM will be taking part in the Northeast Conference on Science and Skepticism – NECSS 2010, April 17th beginning 10:00AM in New York.

There will be a 70 minute panel discussion moderated by John Snyder and featuring David Gorski, Kimball Atwood, Val Jones, and myself – Steven Novella. The topic of discussion will be the infiltration of pseudoscience into academic medicine.

This will be part of a full day of science featuring other excellent speakers, including James Randi, D. J. Grothe, Steve Mirsky, George Hrab, and Julia Galef. There will also be a live recording of the wildly popular science podcast, The Skeptics’ Guide to the Universe.

Go to http://www.NECSScon.org to register.

Anticoagulation is advised for patients who have had a blood clot or who are at increased risk of blood clots because of atrial fibrillation, artificial heart valves, or other conditions. Over 30 million prescriptions are written every year in the US for the anticoagulant warfarin, best known under the brand name Coumadin. Originally developed as a rat poison, warfarin has proved very effective in preventing blood clots and saving lives; but too much anticoagulation leads to the opposite problem: bleeding. A high level of Coumadin might prevent a stroke from a blood clot only to cause a stroke from an intracranial bleed. The effect varies from person to person and from day to day depending on things like the amount of vitamin K in the diet and interactions with other medications. It requires careful monitoring with blood tests, and it is tricky because there is a delay between changing the dose and seeing the results.

In his book The Language of Life, Francis Collins predicts that Coumadin will be the first drug for which the so-called Dx-Rx paradigm — a genetic test (Dx) followed by a prescription (Rx) — will enter mainstream medical practice. FDA economists have estimated that by formally integrating genetic testing into routine warfarin therapy, the US alone would avoid 85,000 serious bleeding events and 17,000 strokes annually.
A recent news release from the American College of Cardiology described a paper at their annual meeting reporting a study of

896 people who, shortly after beginning warfarin therapy, gave a blood sample or cheek swab that was analyzed for expression of two genes — CYP2C9 and VKORC1 — that revealed sensitivity to warfarin. People with high sensitivity were put on a reduced dose of warfarin and had frequent blood tests. People with low sensitivity were given a higher dose of warfarin.

During the first six months that they took warfarin, those who underwent genetic testing were 31 percent less likely to be hospitalized for any reason and 29 percent less likely to be hospitalized for bleeding or thromboembolism than were a group that did not have genetic testing.

Epstein said that the cost of the genetic testing — $250 to $400 — would be justified by reduced hospitalization costs.

At this point, I don’t believe this study. I’ll explain why I’m skeptical.

It seems to me something is wrong with the whole idea of using genetic testing to adjust Coumadin dosage. It doesn’t matter whether a patient has low sensitivity or high sensitivity to warfarin, since the same trial-and-error process of monitoring and dose adjustment will result in a safe dose for either. 

In the typical scenario for a patient who is given warfarin, he is in the hospital where he is first given heparin by injection, usually in the form of low molecular weight heparins like Lovenox. Heparin works by binding to antithrombin and does not require monitoring with blood tests. Warfarin works by an entirely different mechanism. It reduces the amount of vitamin K dependent clotting factors. The heparin provides immediate protection while the warfarin gradually takes effect, a process that lasts for several days and lags behind the blood test used to monitor it: prothrombin time with calculation of the International Normalized Ratio (INR). After the INR reaches the target levels (usually 2-3 times normal), the injectable drug is stopped.

Apparently some providers are in a hurry to get the patient stabilized on Coumadin, so they start with a high “loading” dose and then back off as needed. This may have made sense when patients had to be off the heparin before they could be discharged from the hospital; but today they are commonly taught to give themselves subcutaneous injections and they are sent home, continuing the Lovenox until the Coumadin kicks in. There’s no reason they couldn’t start everyone on a low dose of Coumadin and adjust upwards as needed. Blood tests are done very frequently at first (as often as every day or two) and then can be cut back to as infrequently as once a month if the INR remains stable on a constant dose of warfarin. In the medical facility where I get my care, a Coumadin Clinic directed by a clinical pharmacologist monitors all patients on Coumadin.

A start low/adjust upwards protocol will result in a safe dose for all patients regardless of their DNA. I don’t see how that management plan would be improved by knowing the gene expressions. It takes time to get the genetic test results, and the provider has to start the patient on Coumadin before the test results are even available. Genetics is only estimated to contribute about 70% to dose requirements, and variations in the other 30% could outweigh simplistic dose estimates based on DNA. And testing costs money that I see no need to spend.

It will be interesting to read the actual study when it is published. I will be looking closely for confounding factors that might have influenced results. Could the tested patients have been treated differently in some way because of expectations raised by the DNA test? Did the non-tested patients have optimal management by someone experienced in adjusting Coumadin dosage? Was the study population somehow different from the population the average doctor sees in his practice? How exactly was the Coumadin dose adjusted differently because of the DNA findings? What was the target INR level? Did any of the complications occur in patients who were in the target range? Was compliance assessed? Could alcohol and drug use have affected the INR? The reported 29% reduction of re-hospitalization for bleeding or clotting complications is a relative risk. What was the absolute risk? According to a review article in American Family Physician, the median annual rate of major bleeding in patients on Coumadin ranges from 0.9 to 2.7 percent, and in this study the rate of complications in the highest risk group was only 6.3%. Complications were more likely in elderly patients, patients with co-existing illnesses, and patients with INRs outside the target range. How many of these complications could have been avoided by optimal management?

My common sense tells me patients on Coumadin can be managed just as well without knowing genetic test results. Genetic testing and pharmacogenetics hold great promise for improving individualized patient care decisions, but I don’t think Coumadin is a good example of such benefits. The DNA only confirms what our patient’s response to therapy has already told us.

If there’s one thing about the anti-vaccine movement I’ve learned over the last several years, it’s that it’s almost completely immune to evidence, science, and reason. No matter how much evidence is arrayed against it, its spokespeople always finds a way to spin, distort, or misrepresent the evidence to combat it and not have to give up the concept that vaccines cause autism. Not that this is any news to readers of this blog, but it bears repeating often. It also bears repeating and emphasizing examples of just the sort of disingenuous and even outright deceptive techniques used by promoters of anti-vaccine pseudoscience to sow fear and doubt about vaccines among parents. These arguments may seem persuasive to those who have little knowledge about science or epidemiology. Sometimes they even seemed somewhat persuasive to me; that is, at least until I actually took the time to look into them.

One example of such a myth is the claim that “vaccines didn’t save us,” also sometimes going under the claim that “vaccines don’t work.” The anti-vaccine website Vaccine Liberation has a large set of graphs purporting to show that the death rates of several vaccine-preventable diseases, including whooping cough, diptheria, measles, and polio were falling before the vaccines for each disease were introduced. The the article quotes Andrew Weil:

Scientific medicine has taken credit it does not deserve for some advances in health. Most people believe that victory over the infectious diseases of the last century came with the invention of immunisations. In fact, cholera, typhoid, tetanus, diphtheria and whooping cough, etc, were in decline before vaccines for them became available – the result of better methods of sanitation, sewage disposal, and distribution of food and water.

Bill Maher has said similar things about vaccines, and the “vaccines didn’t save us” gambit has become a staple of anti-vaccine websites. For example, one particularly ignorant blogger wrote:

The mythology surrounding vaccines is still pervasive, the majority of the population still believes, in faith like fashion, that vaccines are the first line of defense against disease. The true story is that nutrition and psychological/emotional health are the first line of defense against disease.

Vaccines are a concoction of chemical adjuvants and preservatives coupled with virus fragments and have clearly been implicated in the astounding rise in neurological disorders around the world, yet the ‘popular’ media has embedded itself as a spokesperson for the pharmaceutical cartel and simply does not report in any responsible way the real situation.

Ah, yes, the “toxins” gambit! Of course, it is true that better sanitation is a good thing and has decreased the rate of transmission of some diseases for which sanitation can do so, many infectious diseases are transmitted person-to-person through the air from aerosolized drops of saliva from coughs and sneezes or from being deposited on objects that people touch frequently, like doorknobs and other fomites.

The “vaccines didn’t save us” strategy is a distortion, as I will show. The best way to demonstrate this is to go on to the very first website that currently shows up on a Google search for “vaccines didn’t save us.” Although the post is from November, it’s the main post that’s been spreading this lie since then. Entitled Proof That Vaccines Didn’t Save Us, it’s one of the most breathtakingly spectacularly intellectually dishonest bits of anti-vaccine propaganda that I’ve ever seen. I say that not because it uses a common anti-vaccine distortion, but rather because it ups the ante by adding a new one clearly designed to address the criticism of the old one. That new distortion hides it in plain sight, too, which is why I have to give the blogger props for sheer chutzpah. Actually, I have to give some backhanded kudos to the person who devised the graphs used in this post, Raymond Obomsawin, PhD. They represent the classic anti-vaccine lie, combined with some very clever cherry picking. I won’t take them all on in this post. Maybe I’ll take some of them on in a future post. In the meantime, what I will do is to take on the first several, because they represent a common anti-vaccine theme that is very similar to the one sounded by the this disingenuous post.

In fact, let’s look at the Vaccination Library claims first. Notice that there are six graphs, four of which are for vaccine-preventable diseases for which widespread vaccination was undertaken, two for which it was not. All of them show decreasing death rates from various diseases. Wow! It seems like slam dunk evidence, doesn’t it? Vaccines didn’t save us! After all, death rates were declining years before the vaccine, and they were declining for the diseases that didn’t even need a vaccine!

Death rates.

Here’s the problem. It’s not surprising that death rates were declining before introduction of the vaccines. Medicine was improving. More importantly, supportive care was improving. For example, take the case of polio. Before the introduction of the iron lung and its widespread use, for example, if a polio patient developed paralysis of the respiratory muscles, he would almost certainly die. The iron lung allowed such patients to live. Some even survived in an iron lung for decades. No doubt improved nutrition also played a role as well. However, if you want to get an idea of the impact of vaccines on infectious disease, take a look at this graph from the CDC of measles incidence, not death rates:

Similar results were seen most recently from several other vaccines, including the Haemophilus influenza type B vaccine, as the CDC points out:

Hib vaccine is another good example, because Hib disease was prevalent until just a few years ago, when conjugate vaccines that can be used for infants were finally developed. (The polysaccharide vaccine previously available could not be used for infants, in whom most cases of the disease were occurring.) Since sanitation is not better now than it was in 1990, it is hard to attribute the virtual disappearance of Haemophilus influenzae disease in children in recent years (from an estimated 20,000 cases a year to 1,419 cases in 1993, and dropping) to anything other than the vaccine.

In the post to which I referred, the most intellectually dishonest graph is this one:

(Click to see full size figure.)

Note how this graph, unlike all the other graphs used to make the claim that “vaccines didn’t save us” actually uses incidence data, in this case from Canada from 1935 to 1983. I was immediately suspicious of this graph, though. The reason should be obvious; the decline in measles incidence is far too smooth. Measles incidence typically varies greatly from year to year. Fortunately, in his chutzpah, Obomsawin included a link to the actual source of the graph. Naturally, I couldn’t resist checking it out, and I found that the link leads to the Canadian Immunization Guide section on the measles vaccine. And this is the actual graph from which Obomsawin allegedly extracted his data:

(Click for full-sized version.)

Note how Obomsawin left out a section of ten years when measles was not nationally reportable. Also note how he has, to be charitable, cherry picked the years to produce the impression of a smoothly declining measles incidence from 1935 to 1959. As I said, it doesn’t get much more intellectually dishonest than that. But it’s even worse than that. The graph above still gives the impression that measles incidence was falling before the ten years for which there is no data. Steve Novella points out that there is a better version of the graph in this reference, and he was kind enough to send it to me, given that, for whatever reason, my university doesn’t have a subscription to the relevant journal:

(Click for full-sized version.)

Note how this graph looks at raw case numbers and shows 40,000 cases of measles in Canada the year before the ten year interruption in the data. All in all, it’s a much clearer representation of the data than the first graph, showing a clear drop that occurred during the ten year period, in the middle of which the measles vaccine was introduced. It also shows another obvious drop in measles incidence later on in the 1990s, when the two-dose measles vaccine program was started. As for why it appears that there is a steep dip in the first graph before the ten year gap, that actually appears to be an artifact. There is no data for 1959, the first year that measles was not nationally reportable in Canada, but the line appears to go to a datapoint at 1959 or 1960. My guess is that whoever made the graph decided to set the value for the beginning of the ten year gap to equal the first datapoint at the end of the ten year gap. In other words, the graph a rather poor representation of the data, and the Canadian government would do well to replace it on its website with something more like the second graph, which makes the point much more clearly.

As intellectually dishonest as Obomsawin’s graph is, this description of Obomsawin matches it:

He has produced academically and/or professionally over eighty-five (85) articles, reports, policy documents, presentations, and publications.

A search of Pubmed reveals only one peer-reviewed publication from 1978, and it’s only a commentary. In any case, apparently served as Director National Office of Health Development of the National Indian Brotherhood (AFN); Founding Chairman of NIB’s National Commission Inquiry on Indian Health; Executive Director in the California Rural Indian Health Board; Supervisor of Native Curriculum, Government of the Yukon Territory; and Evaluation Manager – Department of Indian and Northern Affairs Canada. None of these are scientific positions. More tellingly, he is “currently engaged with government funding as Senior Researcher relative to establishing a Public Sector Policy on Traditional Medicine in Canada.” My translation? He’s somehow managed to get a government grant to try to promote “traditional medicine” in Canada. Apparently, the Canadian government has its own problems with government money going to promote unscientific and pseudoscientific nonsense of the type that NCCAM promotes. In any case, besides Obomsawin’s disingenuous and intellectually bankrupt distortions of incidence data used to serve his apparently anti-vaccine agenda, he has no qualifications to speak of with regard to science or epidemiology that I can find.

It also turns out that Dr. Obomsawin has some other–shall we say?–unconventional beliefs as well. For instance, he is approvingly featured on that aggregator of all things quackery and pseudoscience, Whale.to, where he expresses anti-vaccine views, HIV/AIDS denialism, and admiration for Royal Rife. So what we have here is a woo-meister using cherry picked points on a graph to give a false impression that the measles vaccine was not responsible for the dramatic decline in measles incidence in Canada in the 1960s. Shocking, I know.

Another rebuttal to the idea that vaccines didn’t reduce the incidence of the diseases against which they were designed comes from the simple observation that, as vaccine uptake falls, the disease vaccinated against returns. Always. This is described by the CDC quite well:

Finally, we can look at the experiences of several developed countries after they let their immunization levels drop. Three countries – Great Britain, Sweden, and Japan – cut back the use of pertussis vaccine because of fear about the vaccine. The effect was dramatic and immediate. In Great Britain, a drop in pertussis vaccination in 1974 was followed by an epidemic of more than 100,000 cases of pertussis and 36 deaths by 1978. In Japan, around the same time, a drop in vaccination rates from 70% to 20%-40% led to a jump in pertussis from 393 cases and no deaths in 1974 to 13,000 cases and 41 deaths in 1979. In Sweden, the annual incidence rate of pertussis per 100,000 children 0-6 years of age increased from 700 cases in 1981 to 3,200 in 1985. It seems clear from these experiences that not only would diseases not be disappearing without vaccines, but if we were to stop vaccinating, they would come back.

The United Kingdom is an excellent illustration of this trend. Back in the mid-1990s, it declared measles as under control, thanks to the MMR vaccine. Then came Andrew Wakefield in 1998 with his trial lawyer-funded, incompetent, and possibly even fraudulent study claiming to link the MMR vaccine to “autistic enterocolitis,’ and a credulous, sensationalistic British press to spread his message that the MMR vaccine causes autism. The result was that measles came roaring back in the U.K. to the point that two years ago measles was declared endemic again there.

The Vaccine Liberation graphs and the even more deceptive graphs produced by “Dr.” Obomsawin to claim that vaccine-preventable diseases were already plummeting before the introduction of the relevant vaccines are typical of anti-vaccine arguments. First, they contain enough of a grain of truth to them to sound plausible. After all, better nutrition and better sanitation have in general contributed to better health and contributed to a decreasing toll from various infectious diseases. But they were not enough. Indeed, part of the reason we vaccinated against some diseases is because sanitation wasn’t enough. Was sanitation so much worse in the late 1980s before the Hib vaccine was introduced than it is now? No. Was it probably even that much worse in the 1960s, when the measles vaccine was introduced? Probably not. Yet, such is the myth that the anti-vaccine movement would have parents believe. Such is the intellectually dishonest nonsense they promote.

Why do they do this? J.B. Handley himself has told us why: To bring the U.S. vaccine program to its knees. Or the U.K. program. Or whatever program where the anti-vaccine program has taken hold. The reason is that, no matter how much science says it isn’t, to the anti-vaccine activist, it’s first and foremost always all about the vaccines.