I read a neat article awhile ago and finally got around to writing about it today. The topic of the article is the unfortunate problem of scientific medical studies being wrong and medical treatment being based off of poor quality research. The basic situation is this: Scientists who do medical research need to get their findings published regularly or they are going to be out of a job. Publishing new, exciting findings is drastically easier than publishing boring findings or repetitions of old experiments so scientists are very strongly incentivized to do work that is likely to produce new, exciting results. A big problem with this is most medical studies that show interesting results have only a 0.05 correlation so 5% of them are flat out wrong on the statistics alone. Another issue is that this 5% number relies on the data being perfectly selected, the researcher to exhibit zero bias in recording, analysis or reporting and that the experiment was designed perfectly. In fact none of these three things is usually true and it is certain that the vast majority of medical studies, including those that make the news, have a really substantial chance of being wrong - the article I quoted suggests as high as 80% for normal studies, though that can be improved upon by making the study very random and/or very large, which most are not.
Up to this point we have only talked about errors that are entirely accidental. While it is true that the vast majority of people who get into medical science do so with the genuine desire to discover new, effective treatments rather than to gain influence, money and success there is no denying that people who are worried about their jobs and their credibility end up stretching the truth a lot. There is some dishonesty that is very hard to catch because it is caught up in interpretation and slight variations from proper routine but many studies are tainted by obvious, serious fraud. It is easy to see how it happens... when you are absolutely sure the premise is true and finding it to be true is critical to keeping your job then it seems like only a small crime to tweak the data here and there to prove it to be so. This issue with credibility doesn't end at the researcher though as of course drug companies, homeopaths, health food experts and more make huge amounts of money selling people things that will in theory improve their health. They have every incentive to seize on any new result, proved conclusively or not, and then plug their ears and shout "Buy it! A study said it will make you healthier!" and never, ever look at another study in the field again.
Further yet imagine you are a scientist looking for a new project. You can retest a premise that has been 'proven' in a journal or you can do new research. Retesting means that you either end up saying "Yep, that old study sure was right" which is not interesting or you end up saying "That old study sure was wrong" in which case you manage to make yourself some enemies among those who were betting their careers on it being correct. Even if you are the perfect scientist you have every reason to look for new results that are wrong 5% of the time and leave any flawed study to stand on its own. Medical science has made huge advances but unfortunately almost all of our improved health and longevity can be attributed to very basic things like sanitation, antibiotics, abundant food and reductions in infant mortality. It is critical that we do everything in our power to force a higher standard on medical research so that we can use our medical dollars for the things we *know* help people instead of the myriad of things that we have no reason to think will help.
Vote Pirate!
ReplyDelete