Since the 1960s, in the wake of the thalidomide tragedy, the US has required drug makers to provide “substantial evidence” that drugs are effective. This evidence, says the law, must consist of “adequate and well-controlled investigations.”
Today there is evidence that these standards are being bypassed. This concern is illustrated by the recent approval of the new antibiotic Recarbrio, a drug over which the US Food and Drug Administration had serious doubts. Despite the absence of any clinical studies to provide substantial evidence of its effectiveness the agency approved the drug, a product 40 times more expensive than an existing generic alternative. Did the FDA break its own rules in approving this antibiotic, and what does this case tell us about problems within the agency?
Recarbrio was approved back in July 2019. A Merck drug, it is a three drug combination injectable antibiotic adding relebactam, a new β-lactamase inhibitor drug to imipenem (imipenem-cilastatin), a decades old Merck antibiotic. In pre-marketing studies Recarbrio had been trialled head to head against imipenem: one trial evaluated adults with complicated urinary tract infections (cUTI), and another studied patients with complicated intra-abdominal infections (cIAI).12 Merck’s new combination costs $4000-$15 000 (£3180-£11 920; €3650-€13 700) for a course, which compares with a couple of hundred dollars for the generic version of Merck’s old antibiotic.3
In a press release announcing Recarbrio’s approval the FDA noted its commitment to “the development of safe and effective new antibacterial drugs to give patients more options to fight serious infections.” But the FDA press release contained scant information regarding Recarbrio’s efficacy.4
Merck’s own press release announcing the approval added little detail, summarising the efficacy results in one cryptic sentence: “Approval of these indications is based on limited clinical safety and efficacy data for RECARBRIO.”5 No numbers or data were provided.
Even the drug’s 17 page prescribing information for healthcare providers had little to say about the randomised trial results. In the section of the labelling where such efficacy data are typically presented,6 Recarbrio’s label offered the same information about the cUTI and cIAI studies, saying, “These trials provided only limited efficacy and safety information.”7
Randomised trials are considered medicine’s gold standard. For Recarbrio, however, the FDA emphasised that the data were “limited”—words that experts struggled to explain. When asked to summarise the data that the decision was based on, a principal investigator in Merck’s Recarbrio programme stated, “I think from standard FDA level approvals, imipenem-cilastatin plus relebactam performed very well and had very positive safety and clinical efficacy findings in comparison with comparators.”8
Value added?
However, reviewers at the FDA disagreed. A problem was the patient population. The investigators on Merck’s trial in cUTI described randomly allocating 302 study participants “representative of a wide variety of patients typically seen with cUTI.”2 But in a 299 page FDA memorandum documenting the evidence evaluation of Recarbrio, a team of FDA reviewers concluded that Merck had studied the wrong patient population to evaluate the added benefits of the new drug.
Because the cUTI and cIAI trials did not recruit patients who were lacking treatment options, such as those with imipenem resistant infections, “the contribution of relebactam could not be evaluated in these trials,” the FDA noted. Indeed, laboratory tests showed that the vast majority—more than 85%—of enrolled participants had imipenem susceptible infections.
As the vast majority of patients had infections caused by organisms susceptible to imipenem—the control drug treatment—the concern was that most infections could be successfully treated with the control drug alone, making it impossible to determine whether the combination improved patient outcomes more than the older drug alone. And in neither trial did Merck test the hypothesis that its new drug was more effective than its old one.
Instead of testing the hypothesis of Recarbrio’s possible superiority over imipenem in patients lacking treatment options, Merck’s trials tested a “non-inferiority” hypothesis in patients who already had effective options. “Non-inferiority” is an awkward term indicating that the cUTI trial tested whether adding relebactam to imipenem did not worsen the efficacy of imipenem by more than 15%.
A classic rationale for doing a non-inferiority study is to test experimental interventions that may offer non-efficacy benefits, such as improved safety or greater convenience, as a trade-off for some loss in efficacy.9 Non-inferiority trials are therefore not designed to address the question of improved efficacy over existing drugs but rather to test that any loss in efficacy doesn’t exceed a level deemed clinically acceptable.
The FDA’s review memorandum noted that the agency generally accepts 10% worse efficacy for non-inferiority studies in cUTI. However, Merck chose a 15% worse efficacy cut-off, allowing as many as one in seven patients to potentially have a worse outcome and still declare the trial a success.
FDA: Recarbrio results “as much as 21.3% worse”
But the trial results didn’t even meet Merck’s “15% worse” target. For the cUTI indication, efficacy results showed that Recarbrio “could be as much as 21.3% worse than the control group,” the FDA wrote, as the 95% confidence interval of the results showed Merck’s new drug to be as much as 21% worse in effectiveness than the older and less expensive imipenem. The FDA review states that “these results do not support the NI [non-inferiority] of IMI/REL [Recarbrio] 250 mg group to control [imipenem].”10
Ultimately, the FDA didn’t approve Recarbrio for the types of patients in Merck’s cUTI and cIAI trials. Instead, it approved the drug for patients “who have limited or no alternative treatment options.” (This doesn’t prevent physicians from prescribing Recarbrio “off label” to a wider group of patients.) However, patients with “no alternate treatment options” weren’t studied in any of the trials submitted by Merck.
The lead author of the cUTI trial, Matthew Sims, agreed with the FDA’s decision to limit the drug’s indicated use. He told The BMJ that Recarbrio’s performance against imipenem was not the clinically important question, emphasising that it didn’t make sense to use Recarbrio as a first line antibiotic as was done in the trial. Why, then, did his trial not test Recarbrio’s efficacy in patients with resistant infections? The legal “substantial evidence” requirement, after all, applies to “the conditions of use”—in this case, patients with “limited or no alternative treatment options.”
Sims explained that this wasn’t his call—it was just how things work. “There are certain ways to bring drugs to market through the FDA,” he said. “One of them is a UTI trial, and one of them is an IAI trial . . . and for UTI, it’s a non-inferiority trial against something else that is standard right now. Non-inferiority is the pathway through the FDA.” The cUTI trial Sims participated in was just that: testing Recarbrio in patients who weren’t in need of a new drug because of an expectation that imipenem, the control treatment, would successfully treat their infection. And this trial was necessary, he said, to show that “clinically, the combination does at least as well as the drug alone.”
However, by the FDA’s analysis this trial showed that Recarbrio was as much as 21% worse in efficacy than imipenem.
“Now ultimately,” Sims continued, “you want to use them in these resistant organisms.” He pointed to a third trial in Merck’s submission to the FDA: a small trial of Recarbrio in 50 patients with imipenem-non-susceptible infections. That study was published shortly after Recarbrio’s approval. It compared Recarbrio against colistin plus imipenem in patients who had a wide range of infections. The trial authors concluded that Recarbrio was “an efficacious and well-tolerated treatment option for carbapenem-nonsusceptible infections.”11
However, once again the FDA had serious doubts. Under the heading, “Conclusion of efficacy,” the FDA memorandum stated that the trial was a descriptive study (not hypothesis testing) “and as such provides limited information for an efficacy assessment.” The primary intent of the trial, said the agency, “was to gain some clinical experience” in patients with carbapenem resistant infections. But even then, the FDA noted, “from a microbiologic standpoint, there were very few patients with infections due to CRE [carbapenem-resistant Enterobacterales].”10
Approved anyway, on in vitro studies
So, what about the legal requirement that all newly approved drugs come with “substantial evidence” of their effectiveness?12 For Recarbrio, in consideration of “multiple scientific limitations” in the cUTI and cIAI trials, the FDA concluded that “these studies are not considered adequate and well-controlled.” It also called the third study a “very small,” “difficult to interpret,” “descriptive trial with no pre-specified plans for hypothesis testing.”10 Yet, despite all three clinical studies not providing substantial evidence of effectiveness, the FDA approved Recarbrio.
The drug’s approval attracted little media attention, and the FDA didn’t convene a public meeting of its external advisory committee before licensure, an action the agency often takes when the evidence isn’t clear cut. “Your application for [Recarbrio] was not referred to an FDA advisory committee because outside expertise was not necessary; there were no controversial issues that would benefit from advisory committee discussion,” the FDA’s director of the office of antimicrobial products, Ed Cox, wrote in the agency’s official letter to Merck announcing the approval.
David Ross, associate clinical professor of medicine at George Washington University and a former medical reviewer for the FDA, said that the agency was legally required to hold an advisory committee meeting before approving an application for any new molecular entity unless the agency provided a reason for not doing so. “The FDA’s stated explanation,” he said, “could be applied by the agency to every NDA [new drug application] for an NME [new molecular entity], defeating the whole purpose of this statute.”
Instead of basing its decision on the clinical trials in Merck’s application, FDA’s determination of Recarbrio’s efficacy was based partly on past evidence that imipenem was effective and partly—to justify the new relebactam component—on in vitro studies and animal models of infection, rather than evidence from human trials as required by law.10 Ross pointed out that the FDA’s regulations permitted relying on animal studies as substantial evidence only when human efficacy studies were not ethically permissible, such as drugs to prevent or treat future toxic biological or chemical threats.
Brad Spellberg, an infectious diseases doctor and chief medical officer at Los Angeles County and USC Medical Center, told The BMJ that “the FDA does have the ability to approve therapies under specific circumstances where there is no alternative available and the disease is severe or rare” or where there is “major unmet need.” (Spellberg clarified that he was not speaking on behalf of his employer.)
He said, “I suspect that they approved [Recarbrio] based on a concern for lack of available therapies for CRE, because they hadn’t really understood that we did have therapies for CRE with ceftazidime-avibactam and meropenem-vaborbactam.” But while there was “unmet need” when Recarbrio was being developed, by the time the drug was in phase 3 trials there was no unmet need. “It was already a me-too drug,” said Spellberg.
The FDA rejected this suggestion. It said, “While Avycaz [ceftazidime-avibactam] and Vabomere [meropenem-vaborbactam] had been approved by the time of the Recarbrio approval, treatment options for resistance infections, including CRE, remain limited and may become even more limited in an event of unexpected drug shortages or limited access to a particular drug.”
Spellberg said that the FDA didn’t understand the real world, as “people who don’t practise medicine, don’t practise medicine.” He explained, “There are now six drugs on the market to treat CRE. It is not at all clear what Recarbrio adds to Avycaz and Vabomere, neither of which have been subject to shortages, and both of which are marketed by large pharma companies that are unlikely to go out of business.”
Substantial evidence of effectiveness?
Scott Podolsky, a primary care physician, professor, and director of the Center for the History of Medicine at Harvard’s Countway Medical Library, is concerned that Recarbrio’s approval is essentially a return to a way of regulating medicines that the FDA abandoned a half century ago, before the agency’s “substantial evidence” standard.
He said, “Theoretical expectations, in vitro data, and limited clinical data had been used in the 1950s to justify the approval of such drugs as Sigmamycin and Panalba”—drugs subsequently withdrawn for lack of evidence of effectiveness after the passage of the 1962 drug efficacy amendments to FDA law.13
On Recarbrio, Podolsky said that “there is a certain irony here. For the FDA to state with respect to Recarbrio in 2018 that ‘the applicant has provided substantial evidence of effectiveness of IMI/REL [Recarbrio] and the benefit-risk profile of IMI/REL is acceptable’—at the end of three pages of critiques of the clinical studies used to support such an application—seems in some respects a reversion to the Panalba era.”
He added, “I understand that the context is different: these drugs are essentially being approved for a possible future in which alternatives are absent. But it’s hard not to note the inconsistency of a statement regarding the provision of ‘substantial evidence of effectiveness’ with the wide scope of critiques that had preceded that statement.”
FDA regulations state that only the director of the Center for Drug Evaluation and Research (CDER)—at the time, Janet Woodcock—can waive in whole or in part the FDA’s “adequate and well-controlled studies” approval criteria. “A petition for a waiver is required,” the regulation says.14
While one might expect a waiver to have been created in Recarbrio’s case, a spokesperson for the FDA told The BMJ that “there was no [CDER] director memo in the file” for Recarbrio.
The BMJ asked Woodcock, who today is the FDA’s principal deputy commissioner, if she was aware that the clinical studies of Recarbrio did not provide substantial evidence of effectiveness. She wrote, “No I was not (have not been) aware of this statement in the review memo. Decisions on new drugs are ordinarily delegated several levels down in an organisation as large as CDER. The final weighing of the evidence would normally be documented by the individual signing the action.”
The BMJ asked for confirmation that approvals of new drugs required at least one clinical study of the drug itself that demonstrated substantial evidence—evidence lacking in the case of Recarbrio.
“As I said, I don’t know the facts in this case,” Woodcock replied. Asked whether a waiver from the CDER director was required when the “adequate and well-controlled studies” criteria were not met, she responded, “Well I don’t know, and I’m not working in this area now. Probably best to contact CDER about it.”
A CDER spokesperson told The BMJ in an email that the FDA “applied regulatory flexibility” in approving Recarbrio in consideration of: “(1) the life threatening disease indications, (2) that Recarbrio was shown in in vitro data and animal infection model studies to treat high unmet need bacterial pathogens such as Carbapenem-resistant Enterobacterales, and (3) based on decades of efficacy and safety experience in humans with imipenem-cilastatin (the antibacterial backbone of Recarbrio).”
It’s not clear whether this regulatory flexibility enabled the FDA to conclude that Recarbrio had met the legal “substantial evidence” standard without “adequate and well-controlled investigations” of Recarbrio. The FDA declined to answer the question, saying, “We have no additional information to provide.”