Latest Breakthrough Or Junk Science? How To Tell The Good Research From The Bad
Alright so I may have bitched and moaned through those two incredibly painful semesters of research that I had to do in undergrad and grad school, but at least they did teach me how to interpret research. Sometimes it just so happens that the driest of subjects can be the ones that you end up using everyday, and as an MS nurse/patient I do find myself reading through the latest research almost daily. I often get excited by the over promising titles, only to realize after reading the data and results that the whole study accomplished nothing…whomp whooooomp.
So how do you tell good research from bad research?
Obviously the first step is to look at the source of the study. Was it done by scientists in a controlled study, or was it done by John Smith in his basement? Is it published in a journal, or is it a blog written by someone claiming to have cured themselves by juicing (do they also happen to be selling the product? hmmm suspicious). Can things that aren’t studied in a lab like holistic medicines, therapies, and certain diets be helpful? Of course! But as far as your doctors are concerned FDA approval and solid research data is necessary to recommend the big stuff like strong medications and invasive interventions.
That being said, not every study is going to come up with good results, that’s what the scientific process is all about- trial and error. Scientists form a hypothesis which basically consists of a subject they want to study further, and what they predict the result of the study is going to be. A hypothesis generally goes something like “If you give a mouse a cookie, then he will want a glass of milk”. They then conduct a study where they give mice cookies, and test their desire for milk (actually that sounds pretty fun, lets do this study…). Sometimes the researchers hit the nail on the head and prove their hypothesis right, sometimes the results are nowhere near what they thought they would be (which has actually led to some of the greatest scientific breakthroughs), and most of the time the results are somewhat promising but they show that we need to do more research.
So to find out if the results of a study are “good” we have to ask ourselves two essential questions:
“Is this study valid?” and
“Is this study reliable?”
Validity means that you can say with certainty that there is a cause-and-effect relationship, i.e. the mouse’s consumption of the cookie directly caused their thirst for milk. Maybe they had just watch Got Milk? ad, or perhaps they just always drink milk with their snacks. The point is there are a variety of influences to almost every action, and the goal is to prove that the only possible reason the mice wanted milk was because they had a cookie. This is really tricky to do, especially in healthcare related studies. All sorts of other factors could skew the data such as other medical conditions, the age of the subjects, the diversity and number of subjects, how advanced their disease was, and even whether the researchers and subjects followed every step of the study properly. In many research studies it can’t be said with certainty whether the results show a cause-and-effect relationship or they simply show a coincidence or correlation. This is usually why smaller studies that show promising results have to be followed up by larger trials to establish cause-and-effect. If you can reproduce the results, and ideally if you can reproduce the results on a larger scale, then we can safely say there is validity.
So the major point here is correlation DOES NOT equal causation. Take this ridiculous graph from http://www.tylervigen.com. If you didn’t know how to look at research skeptically you would really believe that eating cheese may cause you to die by bed sheet entrapment!
The reliability of the study speaks to how, well, reliable the data is…pretty straight forward. This usually means that the tools used to collect data were always the same and working properly.
So stay curious my friends, but keep this in mind the next time you read a study on the latest miracle cure or diet. And remember being electrocuted by a power line DOES NOT mean that you will get married in Alabama ;)
