{"id":30843,"date":"2010-11-14T09:15:24","date_gmt":"2010-11-14T09:15:24","guid":{"rendered":"http:\/\/euvolution.com\/futurist-transhuman-news-blog\/of-sbm-and-ebm-redux-part-i-does-ebm-undervalue-basic-science-and-overvalue-rcts\/"},"modified":"2010-11-14T09:15:24","modified_gmt":"2010-11-14T09:15:24","slug":"of-sbm-and-ebm-redux-part-i-does-ebm-undervalue-basic-science-and-overvalue-rcts","status":"publish","type":"post","link":"https:\/\/www.euvolution.com\/futurist-transhuman-news-blog\/medicine\/of-sbm-and-ebm-redux-part-i-does-ebm-undervalue-basic-science-and-overvalue-rcts.php","title":{"rendered":"Of SBM and EBM Redux. Part I: Does EBM Undervalue Basic Science and Overvalue RCTs?"},"content":{"rendered":"

During the most recent kerfuffle<\/a> about whether or not Evidence-Based Medicine can legitimately claim to be science-based medicine, it became clear to me that a whole, new round of discussion and documentation is necessary. This is frustrating because I\u2019ve already done it several times, most recently less than a year ago<\/a>. Moreover, I\u2019ve provided a table of links to the whole series at the bottom of each post…Never mind, here goes, and I hope this will be the last time it is necessary because I\u2019ll try to make this the \u201cgo to\u201d series of posts for any future such confusions.\u00a0\u00a0<\/p>\n

The points made in this series, most of which link to posts in which I originally made them, are in response to arguments from statistician Steve Simon, whose essay, Is there something better than Evidence Based Medicine out there?<\/a>, was the topic of Dr. Gorski\u2019s rebuttal<\/a> on Monday of this week, and also from several of the comments following that rebuttal. Mr. Simon has since revised his original essay<\/a> to an extent, which I\u2019ll take into account. I\u2019ll frame this as a series of assertions by those who doubt that EBM is deficient in the ways that we at SBM have argued, followed in each case by my response.<\/p>\n

First, a disclaimer: I don\u2019t mean to gang up on Mr. Simon personally; others hold opinions similar to his<\/a>, but his essay just happens to be a convenient starting point for this discussion. FWIW, prior to this week I perused a bit of his blog, after having read one of his comments here, and found it to be well written and informative.\u00a0\u00a0<\/p>\n

<\/span>\u00a0What\u2019s in a Name?<\/h3>\n

\u00a0One of Mr. Simon\u2019s objections, in his revision<\/a>, is this:<\/p>\n

\n

What is SBM? Here’s a definition found on the opening entry in the SBM blog:<\/p>\n

“the use of the best scientific evidence available, in the light of our cumulative scientific knowledge from all relevant disciplines, in evaluating health claims, practices, and products.” http:\/\/www.sciencebasedmedicine.org\/?p=1<\/a><\/p>\n

But how does this differ from David Sackett’s definition of EBM?<\/p>\n

“the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.” http:\/\/www.bmj.com\/content\/312\/7023\/71.full<\/a><\/p>\n

The only substantial difference I see is the adjective “scientific” that appears twice in the definition of SBM. The claim on the SBM blog is that EBM ignores scientific plausibility. Actually, ignores is too strong a word.<\/p>\n

“EBM ‘levels of evidence’ hierarchy renders each entry sufficient to trump those below it. Thus a ‘positive’ clinical trial is given more weight than ‘physiology, bench research or “first principles”,’ even when the latter definitively refute the claim.”\u00a0 http:\/\/www.sciencebasedmedicine.org\/?p=42<\/a><\/p>\n<\/blockquote>\n

\u00a0(I agree that \u201cignore\u201d is too strong a word, but I didn\u2019t actually write it that way, as Dr. Gorski pointed out<\/a> and as I think<\/em> Mr. Simon was acknowledging above.)<\/p>\n

A difference between Sackett\u2019s definition and ours is that by \u201ccurrent best evidence\u201d Sackett means the results of RCTs. I realize that this assertion requires documentation, which will come below. A related issue is the definition of \u201cscience.\u201d In common use the word has at least three, distinct meanings: 1. The scientific pursuit, including the collective institutions and individuals who \u201cdo\u201d science; 2. The scientific method; 3. The body of knowledge that has emerged from that pursuit and method (I\u2019ve called this<\/a> \u201cestablished knowledge\u201d; Dr. Gorski has called it<\/a> \u201csettled science\u201d).<\/p>\n

I will argue that when EBM practitioners use the word \u201cscience,\u201d they are overwhelmingly referring to a small subset of the second definition: RCTs conceived and interpreted by frequentist statistics. We at SBM use \u201cscience\u201d to mean both definitions 2 and 3, as the phrase \u201ccumulative scientific knowledge from all relevant disciplines\u201d should make clear. That is the important distinction between SBM and EBM. \u201cSettled science\u201d refutes many highly implausible medical claims\u2014that\u2019s why they can be judged highly implausible. EBM, as we\u2019ve shown and will show again here, mostly fails to acknowledge this fact.<\/p>\n

Finally, Mr. Simon has misinterpreted our goal at SBM:<\/p>\n

\n

But if someone wants to point out that EBM needs work, I’m fine with that. I dislike that they think that EBM needs to be replaced with something better.<\/p>\n

You see EBM as being wrong often enough that you see value in creating a new label, SBM. I see SBM as being that portion of EBM that is being done thoughtfully and carefully, and don\u2019t see the need for a new label.<\/p>\n

I generally bristle when people want to create a new and improved version of EBM and then give it a new label.<\/p>\n

I am as harshly critical of the hierarchy of evidence as anyone. I see this as something that will self-correct over time, and I see people within EBM working both formally and informally to replace the rigid hierarchy with something that places each research study in context. I\u2019m staying with EBM because I believe that people who practice EBM thoughtfully do consider mechanisms carefully. That includes the Cochrane Collaboration.<\/p>\n<\/blockquote>\n

Mr. Simon, we agree! Yes, we are pointing out that EBM needs work. Yes, SBM is that (tiny) portion of EBM that is being done thoughtfully and carefully, and if it were mainly done that way there would be no need to call attention to the point. Our goal is not to change the name of EBM (\u201cgive it a new label\u201d). Our goal is to convince EBM to live up to its current name. Yes, it may self-correct over time, but we are trying to shorten that time. Bad things have unnecessarily happened<\/a>, in part due to EBM\u2019s scientific blind spot: As currently practiced, it doesn\u2019t rationally consider all the evidence. We don\u2019t see much evidence that people at the highest levels of EBM, eg, Sackett\u2019s Center for EBM or Cochrane, are \u201cworking both formally and informally to replace the rigid hierarchy with something that places each research study in context.\u201d<\/p>\n

We chose to call our blog \u201cscience-based medicine\u201d only because the term \u201cevidence-based medicine\u201d had already been taken, and we needed to distinguish ourselves from the inaccurate use of the word \u201cevidence\u201d in \u201cEBM.\u201d I\u2019ve written about this before<\/a>, and have made the point utterly clear:<\/p>\n

\n

These are the reasons that we call our blog \u201cScience-Based Medicine.\u201d It is not that we are opposed to EBM, nor is it that we believe EBM and SBM to be mutually exclusive. On the contrary: EBM is currently a subset of SBM, because EBM by itself is incomplete. We eagerly await the time that EBM considers all the evidence and will have finally earned its name. When that happens, the two terms will be interchangeable.<\/p>\n<\/blockquote>\n

Plausibility Misinterpreted<\/h3>\n

Mr. Simon\u2019s interpretation of our view of plausibility, like that of many others, is wrong:<\/p>\n

\n

I would argue further that it is a form of methodolatry to insist on a plausible scientific mechanism as a pre-requisite for ANY research for a medical intervention. It should be a strong consideration, but we need to remember that many medical discoveries preceded the identification of a plausible scientific mechanism.<\/p>\n<\/blockquote>\n

I think, from his revision, that Mr. Simon understood Dr. Gorski\u2019s explanation of why this was wrong, but I\u2019m not certain. The misrepresentation of scientific plausibility is an issue that I\u2019ve faced for years, as explained previously here<\/a>:<\/p>\n

Plausibility ? Knowing the Mechanism<\/strong><\/p>\n

Let\u2019s first dispense with a simple misunderstanding: We, by which I mean We Supreme Arbiters of Plausibility (We SAPs) here at SBM, do not require knowing the mechanism of some putative effect in order to deem it plausible. This seems so obvious that it ought not be necessary to repeat it over and over again, and yet the topic can\u2019t be broached without some nebbishy South Park do-gooder chanting a litany of \u201cjust because you don\u2019t know how it works doesn\u2019t mean it can\u2019t work,\u201d as if that were a compelling or even relevant rebuttal. Let\u2019s get this straight once and for all: IT ISN\u2019T.<\/p>\n

Steve Novella explained why at the Yale conference<\/a> and again here<\/a>. We talked about it at TAM7<\/a> last summer. For a particularly annoying example, read the three paragraphs beginning with \u201cMr. Gagnier\u2019s understanding of biological plausibility\u201d here<\/a>.<\/p>\n

OK, I\u2019ll admit that I\u2019m beginning to learn something from such frustration. Perhaps we\u2019ve not been so good at explaining what we mean by plausibility<\/em>. The point is not that we don\u2019t know a particular mechanism for homeopathy, for example; the point is that any<\/em> proposed mechanism would necessarily violate scientific principles<\/a> that rest on far more solid ground than any number of equivocal, bias-and-error-prone clinical trials could hope to overturn. The same is true for \u201cenergy medicine<\/a>\u201d and for claims based on non-existent anatomical structures (iridology, reflexology, auricular acupuncture, meridians, chiropractic \u201csubluxations\u201d<\/a>), non-existent physiologic functions (\u201ccraniosacral rhythms<\/a>\u201c), or non-existent anatomic-physiologic relations (\u201cneurocranial restructuring<\/a>,\u201d \u201cdetoxification\u201d with coffee enemas<\/a>, dissolving tumors with orally administered pancreatic enzymes<\/a>). The spectrum of implausible health claims euphemistically dubbed \u201cCAM\u201d is full of such nonsense.<\/p>\n

Reader daedalus2u<\/strong> proposed<\/a> a useful way to\u00a0clarify the point:<\/p>\n

I think the idea of prior plausibility should actually be reframed into one of a lack of prior implausibility. It isn\u2019t that one should have reasons to positively think that something is plausible before testing it, but rather that one should not be able to come up with reasons (actually data) why it is fatally implausible.<\/em><\/p>\n

Some of what We deem implausible will not be fatally so, of course. Implausibility can be based not only on established physical and biological knowledge, but also on studies, as is the case for sticking needles<\/a> into people, injecting them with chelating agents<\/a>, or claiming that autism is caused by childhood immunizations<\/a>.<\/p>\n

EBM, Basic Science, and RCTs<\/h3>\n

Steve Simon wrote<\/a>, \u201cI have not seen any serious evidence of EBM relying exclusively on RCTs. That’s certainly not what David Sackett was proposing in the 1996 BMJ editorial\u2026\u201d And: \u201cNo thoughtful practitioner of EBM, to my knowledge, has suggested that EBM ignore scientific mechanisms.\u201d<\/p>\n

Want serious evidence? Consider these quotations from Cochrane reviews, originally posted here<\/a>:<\/p>\n

\n

In view of the absence of evidence it is not possible to comment on the use of homeopathy in treating dementia.<\/p>\n<\/blockquote>\n

\n

There is not enough evidence to reliably assess the possible role of homeopathy in asthma. As well as randomised trials, there is a need for observational data to document the different methods of homeopathic prescribing and how patients respond.<\/p>\n<\/blockquote>\n

\n

There is currently little evidence for the efficacy of homeopathy for the treatment of ADHD. Development of optimal treatment protocols is recommended prior to further randomised controlled trials being undertaken.<\/p>\n<\/blockquote>\n

\n

Though promising, the data were not strong enough to make a general recommendation to use Oscillococcinum for first-line treatment of influenza and influenza-like syndromes. Further research is warranted but the required sample sizes are large.<\/p>\n<\/blockquote>\n

Yes, EBM undervalues basic science and overvalues RCTs when the former is sufficient to reject a claim.<\/em> EBM also undervalues experimental evidence other than RCTs when the former is sufficient to reject a claim, as will be discussed. Here is how a truly evidence-based review might conclude a discussion of homeopathy for dementia:<\/p>\n

The probability that homeopathy is specifically therapeutic for dementia is, for all practical purposes, zero.<\/p>\n

The following is from my first post<\/a> on the topic, in which I reviewed the overwhelming evidence\u2014from basic science and pre-clinical research\u2014that homeopathic \u2018remedies\u2019 have no, specific therapeutic actions, and wondered why the most esteemed exponents of EBM have written that such treatments are \u201cpromising\u201d and that \u201cfurther randomized trials are needed.\u201d I included the Center for Evidence-based Medicine\u2019s formal \u201cLevels of Evidence\u201d scheme (not copied here), the pertinent quotation from Sackett\u2019s 1996 editorial, my opinion that this failure of EBM was initially unintended, how Sackett et al eventually did address \u201cCAM,\u201d and the Cochrane abstracts quoted above:<\/p>\n

It wasn\u2019t meant to be like this. When I first discussed with my fellow bloggers the curious absence of established knowledge in the EBM \u201clevels of evidence\u201d hierarchy, at least one insisted that this could not be true, and in a sense he was correct. David Sackett and other innovators of EBM do include basic science in their discussions<\/a>, but they recommend invoking it only when there are no clinical trials to consider:<\/p>\n

\n

Evidence based medicine is not restricted to randomised trials and meta-analyses. It involves tracking down the best external evidence with which to answer our clinical questions\u2026And sometimes the evidence we need will come from the basic sciences such as genetics or immunology. It is when asking questions about therapy that we should try to avoid the non-experimental approaches, since these routinely lead to false positive conclusions about efficacy. Because the randomised trial, and especially the systematic review of several randomised trials, is so much more likely to inform us and so much less likely to mislead us, it has become the \u201cgold standard\u201d for judging whether a treatment does more good than harm.<\/p>\n<\/blockquote>\n

That statement is consistent with EBM\u2019s formal relegation of established knowledge to \u201clevel 5,\u201d as seen in the Figure. I am not a historian of EBM and don\u2019t care to be, but I suspect that the explanation for this choice is that \u201cthey never saw \u2018CAM\u2019 coming.\u201d In other words, it probably didn\u2019t occur to Sackett and other EBM pioneers that anyone would consider performing clinical trials of methods that couldn\u2019t pass the muster of scientific plausibility. Their primary concern was to emphasize the insufficiency<\/em> of basic science evidence in determining the safety and effectiveness of new treatments. In that they were quite correct, but trials of \u201cCAM\u201d have since reminded us that although established knowledge may be an insufficient basis for accepting a treatment claim, it is still a necessary one.<\/p>\n

Take note: Sackett wrote, \u201cwe should try to avoid the non-experimental approaches, since these routinely lead to false positive<\/strong> conclusions about efficacy.\u201d My point is that pre-RCT evidence does not routinely (if ever) lead to false negative<\/em> conclusions. In that passage, moreover, Sackett seems to suggest that the only alternative to a \u201cnon-experimental approach\u201d is an RCT; yet there are often other types of experiments that can definitively refute treatment claims, as will be discussed. Eventually Sackett et al did catch wind of \u201cCAM,\u201d but they got it exactly wrong:<\/p>\n

Lacking that perspective, Sackett\u2019s Center for Evidence-Based Medicine<\/a> promulgates an \u201cIntroduction to evidence-based complementary medicine<\/a>\u201d by \u201cCAM\u201d researcher Andrew Vickers. There is not a mention of established knowledge in it, although there are references to several claims, including homeopathy, that are refuted by things that we already know. Vickers is also on the advisory board of the Cochrane CAM Field<\/a>, along with Wayne Jonas and several other \u201cCAM\u201d enthusiasts.<\/p>\n

In another post<\/a> I cited the 2006 Cochrane Review of Laetrile:<\/p>\n

A 2006 Cochrane Review<\/a> of Laetrile for cancer would, if its recommendations were realized, stand the rationale for RCTs on its head:<\/p>\n

The most informative way to understand whether Laetrile is of any use in the treatment of cancer, is to review clinical trials and scientific publications. Unfortunately no studies were found that met the inclusion criteria for this review.<\/em><\/p>\n

Authors\u2019 conclusions<\/em><\/strong><\/p>\n

The claim that Laetrile has beneficial effects for cancer patients is not supported by data from controlled clinical trials. This systematic review has clearly identified the need for randomised or controlled clinical trials assessing the effectiveness of Laetrile or amygdalin for cancer treatment.<\/em><\/p>\n

Why does this stand the rationale for RCTs on its head? A definitive case series<\/a> led by the Mayo Clinic in the early 1980s had overwhelmingly demonstrated, to the satisfaction of all reasonable physicians and biomedical scientists, that not only were the therapeutic claims for Laetrile baseless, but that the substance is dangerous. The subjects did so poorly that there would have been no room for a meaningful advantage in outcome with active treatment compared to placebo or standard treatment… The Mayo case series \u201cclosed the book on Laetrile<\/a>,\u201d the most expensive health fraud in American history at the time, only to have it reopened more than 20 years later by well-meaning Cochrane reviewers who seemed oblivious of the point of an RCT.<\/p>\n

Is that review not serious evidence that the Cochrane Collaboration overvalues RCTs? In this case, moreover, it wasn\u2019t only basic science that Cochrane ignored, but a definitive piece of clinical research that was not an RCT. Sure, I know that Cochrane is not the only pinnacle of EBM, but it\u2019s one of them.<\/p>\n

In both that post and another<\/a>, I called attention to a statement that Edzard Ernst, the most prolific EBM-style \u201cCAM\u201d researcher of the past 20 years, had made in 2003:<\/p>\n

A couple of years ago I was surprised to find that one of the authors of [the Cochrane Laetrile] review was Edzard Ernst, a high-powered academic who over the years has undergone a welcomed transition from cautious supporter to vocal critic of much \u201cCAM\u201d research and many \u201cCAM\u201d methods. He is now a valuable member of our new organization, the Institute for Science in Medicine<\/a>, and we are very happy to have him. I believe that his belated conversion to healthy skepticism was due, in large part, to his allegiance to the formal tenets of EBM. I recommend a short debate<\/a> published in 2003 in Dr. Ernst\u2019s Focus on Alternative and Complementary Therapies<\/em> (FACT), pitting Jacqueline\u2019s countryman Cees Renckens against Dr. Ernst himself. Dr. Ernst responded to Dr. Renckens\u2019s plea to apply science to \u201cCAM\u201d claims with this statement:<\/p>\n

In the context of EBM, a priori plausibility has become less and less important. The aim of EBM is to establish whether a treatment works, not how it works or how plausible it is that it may work. The main tool for finding out is the RCT. It is obvious that the principles of EBM and those of a priori plausibility can, at times, clash, and they often clash spectacularly in the realm of CAM.<\/em><\/p>\n

I\u2019ve discussed that debate<\/a> before on SBM, and I consider it exemplary of what is wrong with how EBM weighs the import of prior probability. Dr. Ernst, if you are reading this, I\u2019d be interested to know whether your views have changed. I hope that you no longer believe that human subjects ought to be submitted to a randomized, controlled trial of Laetrile!<\/p>\n

Uh, talk about “suggesting that EBM ignore scientific mechanisms\u201d! When the principles of EBM and those of a priori<\/em> plausibility clash spectacularly in the realm of CAM, it is a priori<\/em> plausibility that should take precedence\u2014not merely because the latter renders RCTs unnecessary, but because for such questions RCTs tend to confuse rather than clarify, as will be discussed further in the next part of this series.<\/p>\n

I am happy to report that Dr. Ernst wrote me privately about that passage, with the answer that I\u2019d mostly hoped for:<\/p>\n

\n

Have I changed my mind? I am not as sure as the sceptics seem to be that I ever was a supporter of CAM and I am still a bit sceptic about the sceptics [which perhaps makes me the \"ueber-sceptic\"]. Would I argue for more Laetrile studies? NO.<\/p>\n<\/blockquote>\n

Even more to the point, perhaps, is a recent editorial<\/a> by Dr. Ernst in which he calls homeopathy \u201cabsurd\u201d and compares it to other, obvious absurdities, which I doubt he\u2019d have done only a few years ago:<\/p>\n

\n

Should we keep an open mind about astrology, perpetual motion, alchemy, alien abduction, and sightings of Elvis Presley? No, and we are happy to confess that our minds have closed down on homeopathy in the same way.<\/p>\n<\/blockquote>\n

This kind of clear thinking, as easy as it ought to be for intelligent people, seems oddly difficult for those steeped in EBM. I\u2019ll offer another example in part 2, as part of my response to Mr. Simon’s assertion that \u201cThere is some societal value in testing therapies that are in wide use, even though there is no scientifically valid reason to believe that those therapies work.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"

During the most recent kerfuffle about whether or not Evidence-Based Medicine can legitimately claim to be science-based medicine, it became clear to me that a whole, new round of discussion and documentation is necessary. This is frustrating because I\u2019ve already … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"limit_modified_date":"","last_modified_date":"","_lmt_disableupdate":"","_lmt_disable":"","footnotes":""},"categories":[35],"tags":[],"class_list":["post-30843","post","type-post","status-publish","format-standard","hentry","category-medicine"],"modified_by":null,"_links":{"self":[{"href":"https:\/\/www.euvolution.com\/futurist-transhuman-news-blog\/wp-json\/wp\/v2\/posts\/30843"}],"collection":[{"href":"https:\/\/www.euvolution.com\/futurist-transhuman-news-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.euvolution.com\/futurist-transhuman-news-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.euvolution.com\/futurist-transhuman-news-blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.euvolution.com\/futurist-transhuman-news-blog\/wp-json\/wp\/v2\/comments?post=30843"}],"version-history":[{"count":0,"href":"https:\/\/www.euvolution.com\/futurist-transhuman-news-blog\/wp-json\/wp\/v2\/posts\/30843\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.euvolution.com\/futurist-transhuman-news-blog\/wp-json\/wp\/v2\/media?parent=30843"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.euvolution.com\/futurist-transhuman-news-blog\/wp-json\/wp\/v2\/categories?post=30843"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.euvolution.com\/futurist-transhuman-news-blog\/wp-json\/wp\/v2\/tags?post=30843"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}