I have now finished my book, to be called ‘Doctoring Data.’ It has taken a long time to write, mainly because I had to bring together hundreds of different strands of thinking and research. Each strand seemed to get longer and longer as I attempted to pursue them to the end. In many cases I never really found the end.
Some ideas just keep stretching away forever and I had to give up, or else the book would have become a million pages long. And I was told three hundred and ten was to be my limit – or something like that. As if my genius could be contained to a mere hundred thousand words, or so.
Anyway, the main purpose of the book was to look at medical research and data, and try to make some sense of it for those who are interested in looking beyond a medical headline. The book was, at least in part, inspired by a paper written by John Ionnadis.
It was entitled ‘Why most published research findings are false.’ You can easily find it on the internet by searching the title. It is currently the most downloaded paper in recent medical scientific literature
The shortest summary of his paper is, as follows:
‘Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias.’ J Ionnadis.
How can it be, you may think, that most published research is false? Surely research is the one area of human endeavour where bias and dogma are ruthlessly hunted down and destroyed. A scientific finding is a scientific finding….is it not? Did Francis Bacon die in vain?
As Dogbert might say. Hahahahahahahahahahaha!
Or you might be best to pay attention to the quote from Friedrich Nietzsche. ‘There are no facts, only interpretations.’ Of course, he was a bit bonkers, but there is an awful lot of truth to what he said. Especially in medical research. Facts are the most tricky little blighters to get hold of. Interpretation, however, that is stated as fact all over the place
Surely, though, we have ways to ensure that research is pure and objective, such as peer-review. A system of using respected ‘experts’ to check and approve papers before publication. This will weed out papers that are flawed, will it not. Well, here is what Richard Horton (editor of the Lancet) has to say on peer-review:
‘The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.’
There we are, nice and reassuring to know that peer-review is such a fabulous system. As for the quality of published research itself, here is one of my favourite quotes by Drummond Rennie, at the time the Deputy Editor of the Journal of the American Medical Association.:
‘There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too selfserving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.’
A view supported from a slightly different angle by Dr Marcia Agnell, who was the editor of the New England Journal of Medicine for two decades. This was, and remains, the single most powerful and influential medical journal in the world. At least it is, when it comes to citations and impact factor:
“It is simply no longer possible to believe much of the clinical research that is published, or to rely on the judgment of trusted physicians or authoritative medical guidelines. I take no pleasure in this conclusion, which I reached slowly and reluctantly over my two decades as an editor of The New England Journal of Medicine.” Dr Marcia Agnell
Here is a further view on the issue by Richard Smith, editor of the BMJ for many years. He wrote this in his blog:
‘Twenty years ago this week the statistician Doug Altman published an editorial in the BMJ arguing that much medical research was of poor quality and misleading. In his editorial entitled, “The Scandal of Poor Medical Research,” Altman wrote that much research was “seriously flawed through the use of inappropriate designs, unrepresentative samples, small samples, incorrect methods of analysis, and faulty interpretation.” Twenty years later I fear that things are not better but worse…
…The poor quality of much medical research is widely acknowledged,” wrote Altman, “yet disturbingly the leaders of the medical profession seem only minimally concerned about the problem and make no apparent efforts to find a solution.”
Altman’s conclusion was: “We need less research, better research, and research done for the right reasons. Abandoning using the number of publications as a measure of ability would be a start.”
Sadly, the BMJ could publish this editorial almost unchanged again this week. Small changes might be that ethics committees are now better equipped to detect scientific weakness and more journals employ statisticians. These quality assurance methods don’t, however, seem to be working as much of what is published continues to be misleading and of low quality. Indeed, we now understand that the problem doesn’t arise from amateurs dabbling in research but rather from career researchers.’
So, Ionnadis states that most research findings may often be simply accurate measures of the prevailing bias. Current and past editors of the three most respected and powerful medical journals in the world confirm that medical research is warped, biased and flawed and, in many cases simply not believable.
Would this be very evidence used by NICE* to tell us which drugs to use – for example. Why, yes it would be. So be afraid, be very afraid. For an idiotic politician (sorry for the tautology) recently made this announcement1.
‘A Labour government could reduce variation in access to drugs and procedures by making it mandatory for commissioners to follow national clinical guidelines, Andy Burnham has revealed.’ Andy Burnham was, at one time Secretary of State for Health. His is now shadow secretary of state for health. So, if the UK votes for Labour, it will mandatory for all doctors to follow the guidelines based on the evidence that comes from clinical trials.
*NICE stands for the National Institute for Care and Health Excellence. It is supposed to review all evidence for various healthcare areas and decree what is best practice. What NICE says tends to get taken up in many, many, other countries as their views are widely respected and acted upon.