Editor’s note: Three members of the SBM blogging crew had a…very interesting meeting on Friday, one none of us expected, the details of which will be reported later this week–meaning you’d better keep reading this week if you want to find out. (Hint, hint.) However, what that means is that I was away Thursday and Friday; between the trip and the various family gatherings I didn’t have time for one of my usual 4,000 word screeds of fresh material. However, there is something I’ve been meaning to discuss on SBM, and it’s perfect for SBM. Fortunately, I did write something about it elsewhere three years ago. This seems like the perfect time to spiff it up, update it, and republish it. In doing so, I found myself writing far more than I had expected, making it a lot more different from the old post than I had expected, but I guess that’s just me.

In the meantime, the hunt for new bloggers goes on, with some promising results. If we haven’t gotten back to you yet (namely most of you), please be patient. This meeting and the holiday–not to mention my real life job–have interfered with that, too.

The continuum of surgical research in science-based medicine

One of the things about science-based medicine that makes it so fascinating is that it encompasses such a wide variety of modalities that it takes a similarly wide variety of science and scientific techniques to investigate various diseases. Some medical disciplines consist of mainly of problems that are relatively straightforward to study. Don’t get me wrong, though. By “straightforward,” I don’t mean that they’re easy, simply that the experimental design of a clinical trial to test a treatment is fairly easily encompassed by the paradigm of randomized clinical trials. Medical oncology is just one example, where new drugs can be tested in randomized, double-blinded trials against or in addition to the standard of care without having to account for many difficulties that arise from difficulties blinding. We’ve discussed such difficulties before, for instance, in the context of constructing adequate placebos for acupuncture trials. Indeed, this topic is critical to the application of science-based medicine to various “complementary and alternative medicine” modalities, which do not as easily lend themselves to randomized double-blind placebo-controlled trials, although I would hasten to point out that, just because it can be very difficult to do such trials is not an excuse for not doing them. The development of various “sham acupuncture” controls, one of which consisted even of just twirling a toothpick gently poked onto the skin, shows that.

One area of medicine where it is difficult to construct randomized controlled trials is surgery. The reasons are multiple. For one thing, it’s virtually impossible to blind the person doing the surgery to what he or she is doing. One way around that would be to have the surgeons who do the operations not be involved with the postoperative care of the patients at all, while the postoperative team doesn’t know which operation the patient actually got. However, most surgeons would consider this not only undesirable, but downright unethical. At least, I would. Another problem comes when the surgeries are sufficiently different that it is impossible to hide from the patient which operation he got. Moreover, surgery itself has a powerful placebo effect, as has been shown time and time again. Even so, surgical trials are very important and produce important results. For instance, I wrote about two trials for vertebral kyphoplasty for ostoporotic fractures, both of which produced negative results showing kyphoplasty to be no better than placebo. Some surgical trials have been critical to defining a science-based approach to how we treat patients, such as trials showing that survival rates are the same in breast cancer treated with lumpectomy and radiation therapy as they are when the treatment is mastectomy. Still, surgery is a set of disciplines where applying science-based medicine is arguably not as straightforward as it is in many specialties. At times, applying science-based medicine to it can be nearly as difficult as it is to do for various CAM modalities, mainly because of the difficulties in blinding. That’s why I’m always fascinated by strategies by which we as surgeons try to make our discipline more science-based.

Three years ago, I attended a rather fascinating session sponsored by the Society of University Surgeons on surgical research issues involving bioethics and institutional review boards. I realize that there are those out there, particularly on the medical side of things, who with a sneer will refer to the term “surgical research” as an oxymoron, but, trust me on this, there are a lot of surgeons out there trying to do scientifically rigorous studies within the bounds of their discipline to advance the art and science of surgery, all for the good of their patients. (OK, there’s also glory to be had, but the primary motivation still usually boils down to wanting to do good for our patients, with the glory being a secondary motivation.) One other aspect of surgery, besides the ones that I just mentioned, is that surgery is just as much a craft as it is a science. How a surgeon ties knots, where he places the incision and how large he makes it, how he puts together two ends of bowel so that they don’t leak, all of these things are examples where the skill of the individual surgeon matters as much as the scientific validity of what operation the individual surgeon is performing. Surgical research differs from research carried out in medical specialties in that it is much more difficult to standardize operations; placebo controls (i.e., sham operations) are rarely ethically acceptable; and double blind studies are in essence impossible, because the operator will always know what he did and blinding the patient to what was done is similarly impossible without doing sham operations. Moreover, new surgical procedures have a learning curve, such that they may seem less desirable early on in their development but their benefits become apparent the more practice surgeons have at them and as more surgeons learn how to do them well. All of these factors tend to make surgical research, I would argue, in many ways more difficult than medical research, particularly research comparing drug treatments, where double blinding is possible and placebo controls can often be used.

One commonly accepted definition of research is found in federal regulations about human subject research: “Research means a systematic investigation, including research development, testing, and evaluation, designed to develop or contribute to generalized knowledge.” Innovative surgery is often not systematic. Rather, it is designed to benefit individual patients; there is often no intent to publish the surgical series at a later time, or such intent is secondary. In reality, many, if not most, advances in surgery don’t come from careful controlled trials (at least not at first), and the topic of the conference was how to decide when alterations in an established surgical procedure constitute what was referred to as “tinkering” and when they constitute an “innovation,” which is more substantive (more on this later). And when does a surgeon cross the line from tinkering into innovation into actual human subjects research, anyway? These are actually pretty tough questions, with a lot of complexities involved, and the main impetus for the SUS to undertake a project to try to write guidelines about these issues, the preliminary draft of which were presented at the session I atteneded, was as a pre-emptive strike: To do it first before the government steps in and does it in its typically ham-fisted manner. Also, a lot of this tinkering and innovation go on outside the confines of academic medical centers, out in the “real” world of private practice surgeons. The end result was a position statement published in 2008 in the Journal of the American College of Surgeons entitled, appropriately enough, Responsible Development and Application of Surgical Innovations: A Position Statement of the Society of University Surgeons. It begins:

The field of surgery has a unique culture and rich tradition of innovation. Surgeons are trained to perform continuous situational assessment, decision analysis, and improvisation, in preparation for the challenges and creativity required by nearly every clinical case. Indeed, as Riskin and colleagues point out, “most surgeons innovate on a daily basis, tailoring therapies and operations to the intrinsic uniqueness of every patient and their disease.” In addition, unsolved problems and repetitive failures—or at least suboptimal outcomes—stimulate surgeons to strive for a better way to address challenging surgical problems. Consequently, the practice of surgery is steeped in innovation.

In the United States, development of new drugs and medical devices is closely monitored by the federal government. The Food and Drug Administration (FDA) regulates biologics, devices, and drugs by reviewing their safety and efficacy for a given indication before granting approval for marketing and distribution. The labeling (package insert) summarizes what the FDA has deemed the safe and effective use of the product, and “off-label” use of a product, although allowed, may be subject to regulatory scrutiny.

Such oversight does not necessarily extend to surgical innovations. The surgical innovator has historically been allowed to “tinker” with procedures, introducing modifications of varying degrees to the point that a procedure could arguably be called new. In fact, there are numerous examples of such innovations currently in clinical use, most notably, the myriad variations of minimally invasive procedures. Rapidly advancing technology offers the surgeon a steady stream of opportunities to innovate, and this is encouraged by the incentive to publish or present novel approaches, the need to attract new patient referrals, and the innate desire to improve the quality of care delivered by one’s profession. But there are currently no formal regulations that apply to surgical innovations.

Consequently, the gap between regulatory goals and professional reality is widening. With the importance of ensuring patient safety, it is clear that surgical innovation can no longer enjoy its unmonitored status.

Surgical innovation in itself is not problematic. Indeed, a safer, more effective, or less morbid procedure would have a direct and favorable impact on patients. The absence of any organized oversight or mechanism to protect patients from becoming unwitting research subjects is, however, problematic.

Patients may be harmed by innovations that are less effective or more dangerous than expected. More than a decade ago, Roy and colleagues observed, “When large numbers of innovative treatments are being continuously introduced into clinical practice, rigorous testing is mandatory for the protection of individual patients and the just use of limited resources. This holds true with even greater force in light of the evidence that many innovations show no advantage over existing treatments when they are subjected to properly controlled study.” Concerns about inadequate testing of operations have led to increasing calls for vigilance to prevent cavalier innovations and to ensure scientific evaluation of new procedures.

This last paragraph tells what is so problematic about much surgical “innovation.” For example, as I’ve mentioned before, in the late 1980s and early 1990s, laparoscopic cholecystomy rapidly replaced the open cholecystectomy as the standard of care for gallbladder disease requiring surgical intervention. It did so not based on randomized clinical trials showing it to be superior (or at least not inferior) to the prior “gold standard” operation, but rather because the technology was sexy and it was very satistfying (not to mention attractive to patients) to be able to remove a gallbladder while leaving only a few tiny scars and at the same time being able to send the patient home the same day or the next day, compared to a several-day hospital stay. In essence, marketing won out before science, and science has played catch up ever since. Older surgeons had to learn the new technique or watch their bread-and-butter cases be referred to surgeons who could do the new technique. Moreover, in the early days of laparoscopic cholecystectomy, it was documented that the rate of common bile duct injuries was considerably higher than for the old operation. Given that common bile duct injuries can be devastating, even to the point of leading to liver failure, and that they can require huge operations to repair, sometimes with unsatisfactory results, this was not a problem to dismiss lightly. These days, the rate of common bile duct injury for laparoscopic cholecystectomy is very low, but in the “learning curve” phase of the adoption of the procedure it was not and it is arguably still somewhat higher than it was “in the old days” before laparoscopic cholecystectomy. I’ve also written about other examples of surgical procedures that spread like wildfire before they had been adequately proven through clinical trials. The aforementioned kyphoplasty for ostoporotic vertebral fractures is an example of just such a procedure, as well.

Still, there is a gradation between a modification of an old procedure and a whole new procedure. I’ll illustrate with an example. Sid Schwab described a perfect example of tinkering by a private surgeon when he told how, over time, he developed a “mini-cholecystectomy.” (This was before the days of laparoscopic cholecystectomies.) There was no clinical trial to test his hypothesis that he could do a cholecystectomy through a smaller incision. He just did it, confident that he could always make the incision bigger if he was struggling too much. (It’s an axiom in surgery that you can always make the incision bigger, but you can’t make it smaller.) Clearly this qualifies as “tinkering”; i.e., the minor modification of a surgical procedure that does not produce a reasonable expectation of increased risk to the patient. Of course, if Dr. Schwab were too much of a “cowboy” to admit when he couldn’t do the operation using a the smaller incision and started getting into trouble because he couldn’t see or get his instruments in there adequately, you could imagine that this tinkering might have the potential to cause harm. But Dr. Schwab is a prudent and skilled surgeon; it’s reasonable to trust that his judgment is good and that he’ll do the right thing and make the incision bigger if he finds himself struggling too much. Lots of procedures have been improved by such tinkering, and, in fact, often such tinkering is simply a reflection on the skill of the surgeon; for example, one of my partners can do a right colectomy through what to me is an amazingly small incision, while other surgeons use much bigger incisions to do the same operation.

Basically, it was widely agreed that there is generally no need for IRB involvement when tinkering with surgical procedures, as long as informed consent is given or, if the operation was done emergently, the patient is informed afterwards what was done. In practice, though, I can see this as a bit problematic. Consider Sid in the example above. Suppose he went up to his patient and said, “I’m going to do your cholecystectomy, but I’m going to do it a little different. I’m going to use a small incision. It means I might struggle a bit and the operation may take a little longer, but I can always open wider if I have to.” Most patients wouldn’t object, but many would ask: “How much smaller?” A reasonable question. If the incision is only, say, 5% smaller than the usual incision, would it be necessary to tell the patient? After all, only Sid would know for sure that it was smaller than his usual incision. Other surgeons might make that size incision just from their own training, and even Sid, unless he measures all of his incisions to the millimeter, probably varies his incision size by as much as 5% from patient to patient just in the normal scope of his practice.

So, what is innovation? Innovation differs from research in that it is a change in therapy designed to benefit an individual, whereas research is a study according to protocol in which the goal is to gain knowledge, not necessarily to benefit the individual being treated. As the Belmont Report, the basis of all human subjects research regulations in the U.S. states:

By contrast, the term “research’ designates an activity designed to test an hypothesis, permit conclusions to be drawn, and thereby to develop or contribute to generalizable knowledge (expressed, for example, in theories, principles, and statements of relationships). Research is usually described in a formal protocol that sets forth an objective and a set of procedures designed to reach that objective.

When a clinician departs in a significant way from standard or accepted practice, the innovation does not, in and of itself, constitute research. The fact that a procedure is “experimental,” in the sense of new, untested or different, does not automatically place it in the category of research. Radically new procedures of this description should, however, be made the object of formal research at an early stage in order to determine whether they are safe and effective. Thus, it is the responsibility of medical practice committees, for example, to insist that a major innovation be incorporated into a formal research project.

The position statement recommends:

Surgical Innovations Requiring Formal Review

If the innovation is planned, AND:
The surgeon seeks to confirm a hunch or theory about the innovation;

OR:

The innovation differs significantly from currently accepted local practice;

OR:

Outcomes of the innovation have not been previously described;

OR:

The innovation entails potential risks for complications;

OR:

Specific or additional patient consent appears appropriate,

Then:

a) The described review by a local surgical innovations committee is required, plus
b) submission to the national innovations registry is required, and
c) additional informed consent is required of the patient specific to the nature of the proposed innovation.

The recommendations also state that such innovations must comply with FDA and NIH regulations and that informed consent must be obtained, except in the case of emergency surgery, in which case the patient must be informed afterwards, in the case of “unanticipated innovation.” (Of course, one surgeon’s “unanticipated innovation” is another surgeon’s “surgical misadventure.”) It was also recommended that, if all possible, such innovations should be tried out first in an animal before being performed on humans. A further recommendation that was discussed at the SUS meeting but apparently didn’t make its way into the final draft was the creation of a “national innovations registry,” to which surgeons could report their attempted innovations and how they worked. This would be particularly important because most innovations do not work, and some even cause harm. If a surgeon is thinking of trying a new innovation, it would be useful if that surgeon could consult a database to see if someone had tried it before. After all, what’s the first thing we scientists do when considering embarking on a new research project? We hit the scientific literature, of course, to see if anyone has done anything like what we’re thinking of doing before! Scientists have PubMed; many, if not most, innovations, particularly the ones that don’t work, are never published in the peer-reviewed literature. The final recommendation was that “innovation review committtees” be set up to review attempts at surgical innovations. These would tend to be ad hoc committees based on the specific case and far more informal than an institutional review board.

Of course, the distinction between tinkering and innovation (and, for that matter, research) is artificial, and definitely somewhat arbitrary. It all really exists as a continuum. Consider the example of Dr. Schwab’s “minicholecystectomy” again. It’s really a remarkable story of how a surgical procedure, in the hands of a skilled surgeon, evolved over time. Simply by trying to decrease the size of the incision in a systematic fashion, Dr. Schwab ultimately ended up changing his practice. He started using a headlight, special instruments, and a different surgical technique to get the gallbladder out. Ultimately, his consistent tinkering led to innovation, and, I daresay, had it not been for the sudden advent of laparoscopic cholecystectomy in the late 1980’s and early 1990’s, the mini-cholecystectomy that he and others developed would likely have propagated far and wide among surgeons and become the preferred technique for removing gallbladders. In terms of the scientific method, tinkering could be looked at as the stage of hypothesis generation, while innovation could be considered preliminary testing and refining of the hypothesis. The actual research is then the stage where the refined hypothesis is subjected to rigorous testing in a highly controlled, protocolized manner.

Oddly enough, laparosocopic cholecystectomy (LC) is a great example of an innovation that rather got ahead of itself. The potential benefits of the procedure in terms of decreased hospital stay, decreased postoperative pain, and faster return to work were obvious and quickly recognized. However, this procedure, perhaps more than any other, reveals some of the pitfalls in surgical innovation. For one thing, it revealed the effects of marketing and patient demand. Surgeons who learned how to do LCs in those heady early days advertised it heavily, and patients came to see the new procedure as obviously superior to the old before there was much comparative data on complication rates. They started to demand it, and more and more surgeons felt obligated to learn it even before the data from research (formal definition) was all in, lest their patients go elsewhere and their referring physicians abandon them. Eventually, as the research was finally done, it became apparent that there was a considerably higher rate of common bile duct injuries in LC compared to the old method. Granted, it was still a very small number, but the number was nearly ten times smaller still using the boring, old-fashioned open approach to taking gallbladders out, and common bile duct injuries can be devastating, requiring large operations to repair. In the end, it was fairly clear that the benefits outweighed the larger (albeit still very small) risk of common bile duct injuries with LC, but by the time that information was known the procedure had almost entirely supplanted the older procedure. There was no chance to weigh the risk-benefit ratio of LC in anything approaching a calm, measured fashion before introducing the procedure; the analysis was all done after the “cat was out of the bag,” so to speak. Ironically, Sid’s mini-cholecystectomy produced similar good results in his hand to LC; maybe it would have been a better, albeit less sexy, alternative, although it’s always possible that there were complications that would only become apparent if thousands of patients were studied. That’s why innovation needs to be accompanied by research.

There are many problems in considering innovations and research, not the least of which are issues of liability. As one surgeon at the session three years ago asked: “When does ‘innovation’ plus bad outcome equal departure from the standard of care?” The answer was not that encouraging. Innovation by definition is departure from the standard of care, and in the case of a bad outcome it can be a big problem. Surgical innovation committees can ameliorate this problem somewhat by giving cover to the surgeon, as can rigorous informed consent, but clearly in this litigious age the threat of being sued will have a chilling effect on innovation that even these measures may not forestall. Another issue is the composition of these committees. In academic centers, finding the expertise is usually not that big of a problem, but what about in community settings, where either the expertise is lacking or the potential members of such a committee would often be riddled with conflicts of interest, given that potential members would tend to be either partners of or competitors with the surgeon involved? Finally, there is the issue of intellectual property, particularly in the case of devices, even simple ones. Surgeons will be reluctant to share their ideas in a database if they are thinking of patenting them, although it was pointed out that it’s fairly easy to protect one’s idea beforehand by filing a preliminary patent application. And, of course, we must remember that, as much as we hate to admit it, surgeons are human too. Not all of them can always be counted on to act based primarily on what they perceive to be the best interests of their patients.

The bottom line, regardless, is that academic surgeons are going to have to get used to more oversight. Vigorous self-regulation would be the best option if we can pull it off. Gone are the days of the “Wild West,” where “cowboy” surgeons could try whatever innovative idea that caught their fancy. Although many bemoan this new oversight and worry that it will quash surgical innovation, that need not be so if surgeons, rather than the government, act to codify how we distinguish between “tinkering,” innovation, and research and require the level of oversight appropriate to each: IRBs for research and something formalized, but less rigorous, for innovation. Unfortunately, since the SUS guidelines were published in 2008, I haven’t seen much evidence of their having been widely adopted.

On the other hand, what I have seen that is very encouraging is the American College of Surgeons Oncology Group (ACOSOG). Two weeks ago, I attended the Academy of Surgery of Detroit meeting at which Dr. Mitchell Posner, who ran the training program at the University of Chicago when I was a fellow there, presented data on the latest ACOSOG trials. What was most impressive was the effort to which ACOSOG goes to standardize cancer operations rigorously between surgeons. ACOSOG requires surgeons to present videotapes of their operation in order to make sure that the technique is correct and that a surgically adequate resection was performed. In essence, ACOSOG is trying to make surgical trials as close to a randomized, double-blind trial as possible and to remove as many sources of inter-surgeon variation in skill and technique as possible.

In the end, as much as we try to make it a science, surgery can never escape the fact that, by its very nature, it is also a craft. Surgeon skill matters when it comes to patient outcome, and there are often as many ways of throwing a stitch or tying a knot as there are surgeons. Unfortunately, IRBs, as currently constituted, are not sufficiently flexible to take into account tinkering versus innovation versus research. While it is probably not possible to test various new tinkerings or innovations rigorously, it is possible to be a lot more rigorous about overseeing them, preferably before the fact but at least after the fact. Such oversight is increasingly critical as “natural orifice” surgery now breathes down the neck of laparoscopic surgery as the “latest and greatest” fad in surgery. It is also possible to standardize operations as much as possible for purposes of clinical trials. All of these are important developments in the quest to make “surgical science” cease to sound like an oxymoron.