Now, the distinction between a statistical correlation and a causal link is elementary to scientific inquiry - it is, one might say, to research as knowing the alphabet is to writing. In my original posting, I therefore assumed that Coleman had probably controlled for the presence of pre-pregnancy mental health problems, as well as other possible explanatory factors (so-called confounders) in her analysis – reserving my critical comments mainly to biased news reporting of her results. However, subsequent harsh criticism from leading researchers in the field, reported about in another posting, revealed that not only had Coleman not done this elementary footwork, she had made a grossly biased selection of the studies underlying the analysis (with about half of these being of her own or her group and many top-quality studies excluded), included many studies that use deficient methodology (not least those of her own), and – as if that was not enough – undertaken her meta-analysis ignoring several standard ingredients employed and agreed to by researchers world-wide in order to avoid common mistakes and biases. Based on that, I voiced concern about the integrity and quality of the BJP peer review process and editorial policy and speculated about a call for retraction of Coleman's paper (which, in the scientific world means that the content of the article is no longer considered to be valid science). The sub-standard scientific quality pointed out by the critics is reason enough, but since Coleman's conjecture (for this seems what it is) was immediately taken up by anti-abortion extremists world-wide, the editor needs to consider the obvious harm that continued support of the article can have – a harm, due to the scientific flaws of Coleman's work, that is effected for no good reason.
After that, two more critical (so-called "rapid") responses to Coleman's article have been published in BJP. One of these (by David Ferguson, John Horwood and Joseph Boden) attempts to defend Coleman, mainly by attacking one of the critics. However, the very serious criticism voiced in the other responses is simply ignored. Moreover, it is obvious that Ferguson et.al. seem to be as oblivious to the elementary distinction between statistical correlation and causal linkage as is Coleman. They write:
....despite the claims made in previous reviews about the absence of association between abortion and mental health, when data are pooled across studies there is consistent evidence suggesting that women having abortions are at modestly increased risks of mental health problems when compared with women coming to term with unplanned/unwanted pregnancies.Note here that the expressions are "association between abortion and mental health" and "increased risk". Does this support Coleman's conjecture about a causal link? No, it does not, as everyone who has ever taken a primer in statistical method knows. In fact, "association" and "increased risk" means the same, and this meaning is that there is a statistical correlation between having an abortion and suffering from subsequent mental health problems that is stronger than that between carrying a pregnancy to term and suffering from subsequent mental health problems. Yes, you read rightly, the same statistical correlation that is generally agreed on by everyone. Ferguson et.al. either seems to have mistaken what the issue worthy of inquiry is from the outset, or missed some crucial lessons of science class. In short, this response is a misnomer and completely irrelevant to the debate on Coleman's conjectured findings.
So, the basis for assessing Coleman's article as ripe for retraction seems to stand. Not surprisingly then that just such a call was today sent to BJP. I happened to come to know about it through one of the internet forums of my own field of research (the Bioethics International group on Facebook), asked the author of the call to see it, and received his permission to quote it verbatim. It will still take a few days for BJP to decide whether or not to publish it among the "rapid responses", but until then (or in case the editor should chose to try to avoid the embarrassment of publishing it), you can read it here in its entirety:
Coleman Article Should be Retracted, Not Debated in a Subsequent Issue of BJP
Serious flaws in the reporting and conduct of the Coleman review should have been identified in pre-publication review and not left for readers of BJP to sort through subsequently. The article should be retracted and should not given the dignity of post-publication debate in a subsequent issue of the journal.
The review lacks the fundamental transparency that is expected of systematic reviews and meta-analyses and needed to allow readers to independently evaluate its conduct and interpretation of results without having first to go back to the original studies. Search strategies are not even provided in sufficient detail for readers to ascertain the adequacy and completeness of the retrieval of relevant studies.
Results for 36 effects obtained from 22 studies that are integrated into a single effect size represent highly diverse outcomes ranging from smoking of marijuana to suicide. The overall effect size that is calculated does not generalize back to the individual outcomes in any meaningful way. This aspect of the meta-analysis recalls a photo often incorporated into workshops on meta-analysis. The photo depicts the famous road sign for New Cuyama, California in which a total of 4663 is indicated for a population of 562, an elevation of 2150 feet, and a date of establishment of 1951. The calculation of an estimate of the heterogeneity of the effect size reported by Coleman is missing, in violation of standards for reporting a meta-analysis.
Multiple effects sizes are obtained from individual studies are integrated in a way that violates basic assumptions of independence of individual effect sizes that are required for a meaningful meta analysis. The 22 studies include 13 from Coleman's author group, and so the meta analysis violates usual expectations that a meta analysis be independent of the author group who generated the original studies. David Reardon who is a co-author of Coleman on a number of these studies has declared his strategy :
"For the purpose of passing restrictive laws to protect women from unwanted and/or dangerous abortions, it does not matter if people have a pro-life view...In some cases, it is not even necessary to convince people of abortion's dangers. It is sufficient to simply raise enough doubts about abortion that they will refuse to actively oppose the proposed anti-
abortion initiative. In other words, if we can convince many of those who do not see abortion to be a 'serious moral evil' that they should support anti-abortion policies that protect women and reduce abortion rates, that is a sufficiently good end to justify NRS efforts. Converting these people to a pro-life view, where they respect life rather than simply fear abortion, is a second step. The latter is another good goal, but it is not
necessary to the accomplishment of other good goals, such as the passage of laws that protect women from dangerous abortions and thereby dramatically reduce abortion rates."
Many of the studies included in the Coleman meta-analysis, including most of the studies conducted by her group, are strongly criticized by other researchers and excluded from consideration in other systematic reviews, including a forthcoming report by the National Collaborating Centre for Mental Health (NCCMH) at the Royal College of Psychiatrists
(RCPsych). One can only speculate on the timing of the BJP's publishing of Coleman's review relative to the impending release of the RCPsych report. Results of some of the original Coleman studies are not replicated in subsequent re-analyses of the same data sets by others. Coleman integrates results from studies without controlling for measures of mental health outcomes obtained prior to an abortion and in a number of instances, the mental health outcomes entered into her meta analysis were obtained before the abortion. In other instances, the effects reflect differences between women who obtained an abortion for an unwanted pregnancy versus women who delivered a wanted baby, a grossly
inappropriate comparison if the intention is to obtain a valid estimate of the effects of abortion on mental health.
It is a mater of technical details, but important to evaluating Coleman's meta analysis that she used the wrong formula to calculate population-attributable risk and violated basic assumptions for such a calculation.
These serious flaws were apparent in a cursory reading of Coleman's article. I am confident that a closer read and a retrieval of the original studies and others that were ignored by Coleman would have yielded still more problems. But I think this analysis reaches the threshold for demonstrating the necessity of retracting the Coleman article and it begs an explanation for the nature of the peer review that led to the article being accepted.
The Coleman article is not a contribution to scientific literature but rather represents the revenge of Coleman and her offer group on the scientific community which has held their work to basic objective scientific standards, criticized its poor quality, and excluded from
integration into systematic reviews on the basis of objective criteria.
1. Reardon DC (2002). A defense of the neglected rhetorical strategy (NRS). Ethics Med 18 (2): 23-32.
James C. Coyne, Ph.D.Careful as I am, not being an expert in this field, I still must say that the combination of the massive sound criticism against Coleman's analysis and methodology, as well as the obvious fault that I can spot myself, should have the editor make the retraction move. The obvious ideological agenda underlying the research of Coleman and her team, pointed out also in another of the rapid responses, adds to this, since it makes probable the hypothesis that the faults of Coleman are not (only) the result of incompetence, they are deliberately engineered to achieve the shallow appearance of publicisable scientific claims in order to give the political anti-abortion movement its 15 minutes fame.
Director, Behavioral Oncology Program
Abramson Cancer Center and
Professor of Psychology
Department of Psychiatry
University of Pennsylvania School of Medicine