Origins of Snake Oil: The truth about the most famous retracted COVID-19 study

Aug 20, 2020

E. Rosalie Li, interdisciplinary public health Johns Hopkins Bloomberg School of Public Health

The controversial study gave hope to the world. That was before scientists discovered the trial contained worrisome flaws. Whether the drug helped people heal faster, this study never had the potential to tell us.

Upon reviewing the paper, one finds numerous errors so extreme that it is difficult to understand how it got published. How could this research have undergone peer review and emerged in such a state?

This is where the story lies. Ultimately, the study revealed a profound truth: studies may be so flawed as to defy belief in their publication and still mislead the public as if the results had been accurate.

The Impossible Study

An ethics committee green-lighted the 14-day study on March 6th, 2020. If the trial began the next day, then March 20th was the earliest date they could have finished. Researchers presented early trial results, yet to be reviewed by outsiders, for the first time on March 16th via YouTube.

Also, on March 16th, the team submitted the findings to the International Journal of Antimicrobial Agents. The journal accepted by March 17th, publishing on March 20th. This left a one-to-two-day period for the peer-review process, which may ordinarily take months.

The strangest aspect of this paper's journey to publication may be that, once the results were uploaded to preprint servers where anyone could access them, a more thorough external review did not appear to have occurred.

Over 40,000 COVID-19-related preprint studies have been accepted quickly, sought review, and published formally a month or two later with the needed corrections.

Peer review involves having someone with expert-level knowledge read and critique a paper. Ideally, this person would disagree with you, as they would be more likely to spot weaknesses in your arguments. Asking someone outside the field would be akin to asking a classical pianist to diagnose strange noises from your car. Reviewing your own work would be like grading your own homework.

If peer-review happened as the journal claims, the job was so poor as to be undetectable. That isn’t said metaphorically. The paper submitted for review shows no differences from the original.

Scientists reviewed the document before it was published and compared the earliest version to the journal’s published version. The earlier draft came from a Google Drive version circulated among scientists before March 16th.

They had this to say:

“These versions of the study report were the same as the one we reviewed, indicating no or limited external peer review for the final published version.”

The International Society of Antimicrobial Chemotherapy stated in April 2020:

“ISAC shares the concerns regarding the above article published recently in the International Journal of Antimicrobial Agents (IJAA). The ISAC Board believes the article does not meet the Society’s expected standard, especially relating to the lack of better explanations of the inclusion criteria and the triage of patients to ensure patient safety.”

The ISAC concluded:

“it is important to help the scientific community by publishing new data fast, this cannot be at the cost of reducing scientific scrutiny and best practice.”

The concrete evidence shows that nothing changed. Was the paper given to someone uniquely unskilled in peer review? The evidence suggests not.

Conflict of Interests Galore

The Editor-in-Chief of the publishing journal is also the author of the study. The issue featuring the research paper also included an editorial penned by the study authors, including one Editor-in-Chief. The letter would give a sparkling review, making statements not reasonably made from the results of a single study, even if the drug had helped people (sic):

“It’s possible use both in prophylaxis in people exposed to the novel coronavirus and as a curative treatment will probably be promptly evaluated by our Chinese colleagues. If clinical data confirm the biological results, the novel coronavirus-associated disease will have become one of the simplest and cheapest to treat and prevent among infectious respiratory diseases.”

— Chloroquine for the 2019 novel coronavirus SARS-CoV-2 by Colson, Rolain, and Raoult

Had the results confirmed, as the quote states, it could not have shown the drug would “become one of the simplest and easiest to treat and prevent among infectious respiratory diseases.”

The study as it was, and not as the authors presented it, never had the potential to deliver firm answers. Partly, the controversy with this study comes from discussing two separate issues as one:

1. The efficacy of hydroxychloroquine and
2. The quality of this study.

Missing Patients: Method in the Madness

The final draft shows 20 patients receiving treatment, six less than at the beginning. Looking at the data analysis, one would assume that the missing six patients dropped from the trial. When people drop from a trial, we “censor” the data, which stops dropouts from influencing the results.

That is not what happened.

Three ended up in the ICU, one died, one could not tolerate the side effects of treatment, and the last person left the hospital. The authors analyzed the study as if these critical or dead patients had stopped taking the drug.

The percentage of people who died or were hospitalized is almost exactly what one expects to see without treatment, with 15 to 20% severe cases and a 1% death rate overall. Effectively, all patients who conflicted with the authors’ conclusion ended up excluded from the data. Whether that was the intention is another issue entirely, but it was the result.

The study bases much of its claims on results taken from Day 6 of treatment, but that was different than a planned checkpoint for the study. They switched endpoints, citing the promising nature of the results. That’s a problem.

The PLoS Clinical Trials Editorial Board highlighted endpoint switching.

“A fundamental principle in the design of randomized trials involves setting out in advance the endpoints that will be assessed in the trial, as failure to pre-specify endpoints can introduce bias into a trial and create opportunities for manipulation.”

The clinical trial registry says they planned to check patients on Day 1, Day 4, Day 7, and Day 14. Day 6 appears outside the planned days. It begs what happened on Day 7. The paper was submitted on March 16th, so it’s reasonable to expect them to include that in the preprint.

The results appeared like a control group took part, but one author openly doesn’t believe in controlled trials. Some patients assigned themselves rather than random sorting and received treatment at different locations. Any differences might explain why one group fared better.

The paper claims that all patients were older than 12, but the report appears to have several under that age. Some patients go negative before positive again. It’s unclear how this test shows anything about the drug’s ability to help. The authors neglected to describe these aspects fully and show no other factors could explain the fantastic results.

COVID-19 patients are already at risk for acute cardiac injury, myocarditis, and cardiac arrhythmias — all heart-related afflictions.

In response, Dutch scientists detailed a litany of valid and serious concerns. “Even if a larger randomized controlled trial (RCT) showed that the combination of (hydroxy)chloroquine and azithromycin would be effective in patients with COVID-19, safety would still be an issue.” The criticism isn’t strictly American.

The toxic potential of the drug — in all things, the dose makes the poison — increases the risk of death. While it may not have mattered much in malaria patients, it matters for COVID-19 patients whose hearts may be directly infected.

According to a JAMA cardiology report, “In patients with coronavirus disease 2019 (COVID-19), cardiovascular involvement occurs frequently.”

The Promised Cure

It took less than a day for the too-good-to-be-true claims to catch the eye of elected officials and a world desperate for a cure.

Officials and the public may not have understood the problems in the study. The public heard that a cure was soon to come, leaving scientists trapped in the difficult position of delivering the bad news. This highlights scientists' responsibility to honestly represent their work and for politicians to defer analyzing scientific results to qualified parties.

The authors have dismissed studies to the contrary as “crappy” while not applying those same design standards to their study. At least one author has a history of outright fabricating data, and insiders have claimed that fear of that author has led to silence.

The dogmatic claims from the authors, taken up by political figures, quickly polarized public opinion on a situation that even those who study the field would need time to understand fully.

Quickly, many dismissed scientists who raised valid concerns as politically motivated, but the evidence for that has yet to be supplied. One cannot dismiss that which conflicts with their chosen perspective by taking any plausible rationale and passing it off as fact. On the contrary, a welling sea of evidence shows the fears were justified. That appears to be the root of this story.

Upton Sinclair once noted:

“It is difficult to get a man to understand something when his salary depends upon his not understanding it.”

Clinging to the belief that a cure was within our grasp has much more appeal than that the study authors misled us from the bottom to the top. Whatever the truth of the study’s results, it no longer mattered once media outlets ran with it. The ill-fated effort to help people understand centered on evidence, but there could be no sufficient evidence to the contrary for some. Scientists failed to see that opinions that pass without evidence likely won’t be dissuaded by more of it.

Reliable, unreliable—it’s all the same.

Monolithic public pressure meant scientists had to repeat the studies ad nauseam. A lack of coordination and centralized response meant “the effort [was] marked by disorder and disorganization, with huge financial resources wasted.”

Whether the drug had been helpful, a single study could not and should not be used to draw that broad conclusion. There are too many potential variables, even with thorough research. For the maybe-wonder-drug hydroxychloroquine, follow-up studies boomed out of control. STAT reported in July 2020 that 1 in every six drug trials was chloroquine-related.

That means fewer resources went to other potential therapies, a loss felt more strongly now that hydroxychloroquine appears to increase mortality and mixed results on benefits.

Evidence had little to do with what followed the media and political frenzy. We fulfilled precisely that which we knew would happen and could have avoided.

Kai Kupferschmidt, a noted science journalist and molecular biologist, wrote in a book from 2018, “New plagues will come. Whether they will lead to catastrophe, will depend mostly on what wins out in the end: empathy and innovation or ignorance and selfishness.” With that, few could disagree, but I would add that humility seems to underpin success or failure.

The Prophetic Warning

In 2014, the Director-General of the WHO gave a keynote speech that was nothing short of prophetic. Yet another outbreak had just shaken the heads of state who now cared to hear the messages they had long ignored, messages that might have saved their countries from the devastating loss they now faced.

“I have never seen an infectious disease contribute so strongly to potential state failure… but I will use the outbreak to show… some key arguments that WHO has been making for decades.

First, the outbreak spotlights the dangers of the world’s growing social and economic inequalities. The rich get the best care. The poor are left to die.

Second, rumors and panic are spreading faster than the virus. And this costs money.

Third, when a deadly and dreaded virus hits the destitute and spirals out of control, the whole world is put at risk.

Fourth, decades of neglect of fundamental health systems and services mean that a shock, like an extreme weather event in a changing climate, armed conflict, or a disease run wild, can bring a fragile country to its knees.

You cannot build these systems up during a crisis. Instead, they collapse… Finally, the world is ill-prepared to respond to any severe, sustained, and threatening public health emergency.”

The Director-General mentioned in this address the World Bank’s shocking estimation that 90% of economic losses in a pandemic come from irrational and uncoordinated efforts.

The U.S. response to the crisis has been nothing if not irrational and uncoordinated, though we could have avoided it. Our results tell the story of our choices and the predictable reasons we made them.