How to Evaluate a Study
Not all studies should be treated equally. Below are a few key factors to consider when evaluating a study’s conclusions.
- Has the study been reviewed by other experts? Peer-review, the process by which a study is sent to other researchers in a particular field for their notes and thoughts, is essential in evaluating a study’s findings. Since most consumers and members of the media are not well-trained enough to evaluate a study’s design and researcher’s findings, studies that pass muster with other researchers and are accepted for publication in prestigious journals are generally more trustworthy.
- Do other experts agree? Have other experts spoken out against the study’s findings? Who are these other experts and are their criticisms valid?
- Are there reasons to doubt the findings? One of the most important items to keep in mind when reviewing studies is that correlation does not prove causation. For instance, just because there is an association between eating blueberries and weighing less does not mean that eating blueberries will make you lose weight. Researchers should look for other explanations for their findings, known as “confounding variables.” In this instance, they should consider that people who tend to eat blueberries also tend to exercise more and consume fewer calories overall.
- How do the conclusions fit with other studies? It’s rare that a single study is enough to overturn the preponderance of research offering a different conclusion. Though studies that buck the established notion are not necessarily wrong, they should be scrutinized closely to ensure that their findings are accurate.
- How big was the study? Sample size matters. The more patients or subjects involved in a study, the more likely it is that the study’s conclusions aren’t merely due to random chance and are, in fact, statistically significant.
- Are there any major flaws in the study’s design? This is one of the most difficult steps if you aren’t an expert in a particular field, but there are ways to look for bias. For example, was the study a “double-blind” experiment or were the researchers aware of which subjects were the control set?
- Have the researchers identified any flaws or limitations with their research? Often buried in the conclusion, researchers acknowledge limitations or possible other theories for their results. Because the universities, government agencies, or other organizations who’ve funded and promoted the study often want to highlight the boldest conclusion possible, these caveats can be overlooked. However, they’re important when considering how important the study’s conclusions really are.
- Have the findings been replicated? With growing headlines of academic fraud and leading journals forced to retract articles based on artificial results, replication of results is increasingly important to judge the merit of a study’s findings. If other researchers can replicate an experiment and come to a similar conclusion, it’s much easier to trust those results than those that have only been peer reviewed.