New evidence gives supporters of chiropractic a headache

A paper was published and much discussed online recently, which demonstrates all the problems that I - and other critics - have with the way research is done and interpreted in the world of chiropractic.

The study looked at the effect of chiropractic neck manipulation on people who have migraines.

On the face of it, the article concerned looks like a pretty fair and well conducted study. Despite the methodological difficulties of randomisation and blinding of participants with manual therapies, a genuine effort was made as part of the study design to allow for this.

The trial was a three-armed study comparing chiropractic spinal manipulation (CMST) with a sham manual therapy and a group who continued with their usual care.

Although the deliverers of the manual therapy would know what they were providing, a level of blinding to treatment allocation was possible. It was refreshing to see as well that they had checked to confirm that the blinding of subjects was maintained throughout the study. This is extremely important in assessing the validity of a study, and as the authors point out

The importance of our successful blinding is emphasized by the fact that all previous manual-therapy RCTs on headache lack placebo._

The outcome measures chosen were reasonably fair and representative of the group studied. The statistical analysis was conventional enough. They had performed power calculations using a reasonable comparator, which again increases confidence in the validity of the results and shows they were taking the methodology seriously.

The robustness of the methodology is likely the reason it was included in the European Journal of Neurology – a solid, second-tier journal with a credible reputation.

I would not quibble with the summary of their conclusion, published in the paper’s abstract, that said:

It is possible to conduct a manual-therapy RCT with concealed placebo. The effect of CSMT observed in our study is probably due to a placebo response.

This is the correct scientific interpretation of the data. If it was a drug trial, we would conclude there was no pharmacologically relevant effect within the parameters shown, and it would be considered a negative study.

How very different then is the analysis of the same study by chiropractors. The Chiropractors’ Association of Australia (CAA) mentioned it in their press release titled saying “chiropractors welcome latest evidence of headaches”. The relevant quote is:

A paper published in the European Journal of Neurology in September 2016 was the latest in a series of papers to confirm the effectiveness of chiropractic health care in treating people with migraines. The study of 104 people in Norway found that Migraine symptoms were substantially reduced as a result of chiropractic treatment.

Critical reporting of this study, such as found on the website of the American Council on Science and Health, throws up some interesting conversations and interpretations in the comments section.

While I don’t generally read online comment sections (apart from this column naturally) it’s worth making the effort in this particular case. The author of that piece has made the same basic epidemiological arguments as I have above, and come to the correct conclusion that it is a negative study.

One of the commenters accuses the author of deliberate bias and makes the following piquant observation:

To show how much spin this “article” has the title could have been: Have a Migraine? Skip the meds, sham and Gonstead CSMT both effective. more than medical care.

This commenter appears to be making the same error of interpretation as the CAA. The point of a three-armed study is to differentiate between the effect of any intervention in the study (due to placebo responses) and the improvement due to simply being observed in an artificial situation (known as the Hawthorne effect).

If a treatment is genuinely efficacious, one expects to see three divergent curves with: a minor improvement in the no-intervention group; a larger improvement in the sham intervention group; and a clearly larger improvement in the true intervention group.

A treatment which lacks effectiveness will produce results in which the two intervention groups are indistinguishable. That is exactly what this study returned. Observe the graphs for yourself.

Note carefully that the authors of the original study are careful to claim the only conclusion that can be drawn is that it is feasible to conduct a manual therapy study in a single-blinded fashion.

They made no claims about the efficacy of the intervention apart from the fact it was equivalent to sham, and in fact are explicit about the fact they believe the treatment effects were all placebo. Outright claiming the opposite is a new peak of disingenuousness for supporters of chiropractic.

Another characteristic of treatments which do not have efficacy is that the more rigorous the study, the more the claimed effect disappears into the statistical noise. To be definitive about a lack of efficacy, a much larger study would be needed.

The results above would not inspire me to spend a couple of million dollars on a study with 200 people in every arm. Results like this over the years have killed off hundreds of medications which were being developed by Big Pharma.

Academic chiropractors are in the invidious position of trying to establish that an ideologically-based treatment system has a scientific basis. They should be careful what they wish for when conducting rigorous studies, as they may find their fondest beliefs being dashed on the rocks of reality.