4 Tips for Critically Evaluating Data in the Media
A quick Google search will yield millions of results for the importance of critical thinking for students, individuals in the workplace, and, simply, as human beings existing in this world. News articles often make claims based off of and present scientific data (especially now in the wake of COVID-19), but how often do we pause and think critically about the data they’re presenting? Confirmation bias is a tendency to be more responsive and sensitive to information that aligns with our pre-existing beliefs (Nickerson, 1998). This can lead us to accept claims allegedly driven by data because they already fit into how we see the world. The flip side of this is instantly rejecting claims and reputable data because we disagree with the conclusion given or the conclusion challenges or contradicts our beliefs. So, how can we safeguard ourselves (and our friends and families) from falling into this trap? Here are some tips for you (and for your family and friends who may be less familiar with interacting with scientific data).
1. Be wary of causal language in headlines and read the full article.
As researchers, we know the ability to make causal assertions in scientific studies is ultimately limited. However, we can still be swayed by flashy headlines; in reading the full article, we may notice that that reporters are extrapolating beyond what the researchers found. So, let’s return to the basics: it’s hard to determine causality. Survey studies and observational studies provide us with important information about whether variables are related and lay the foundation for predicting the nature of relationships, but they do not give us information about causality. In order to truly determine cause and effect, researchers need to have tight control over their study to rule out possible alternative explanations for the causal relationship they observe. This tends to involve random assignment of participants to experimental conditions and careful manipulation of one variable at a time. This information cannot be gleaned by a headline alone. We get a better sense of how the data was collected and therefore how much leeway we have in talking about the data by reading the full article.
2. Spend time looking at the graphs and figures presented.
Figures can be misleading. Pay attention to the vertical scale (i.e., the Y-axis) and note whether the scale starts somewhere other than zero, skips numbers, appears stretched out (i.e., too big) or shrunk down (i.e., too small). Changes in the scaling can result in vastly different images. See the example images below (generated from fictious data). It’s also important to focus on whether the data in the graphs and figures represents raw data, averages, percentages, estimated means, or future predictions. Unless we slow down to think critically about the type of data we’re looking at, we may risk drawing erroneous conclusions.
Notice how in the first graph the differences between conservatives, moderates, and liberals seem quite drastic compared to the second graph. However, the only difference between the two graphs is the scaling along the Y-Axis. Also, the second graph indicates that we’re looking at group averages for endorsement of Policy A and that response options ranged from 1-10.
3. Check your own biases.
Social media news feeds are a key news source in today’s world. However, the news articles that appear in social media feeds are a combination of the actions of editors and journalists producing the content and site-driven algorithms which rely upon users’ previous interactions on the site and the activity of the members in their social network (Fletcher & Nielsen, 2019). As a result, we can be more likely to see articles that already align with our worldview. When news articles appear in our feeds, we should ask ourselves a few questions:
- Who is sharing this information? Whether it’s someone you know shares your worldview or someone you know whose worldview differs drastically, our feelings about that person and their view can lead to us rushing a judgment about the news article they’ve shared. Slowing down to think about how you feel about this person may allow you to be thoughtful about the article they’re shared and its conclusions.
- What website does this article come from? Not all news sites are created equally. Pay attention to the article’s host website and whether the piece is an investigative report or an opinion/editorial.
- Does this evoke a strong emotional reaction in me? If so, why? Emotional reactions heavily influence our decision making (e.g., Kahneman, 2003; Weber & Johnson, 2009). So, the emotions we experience as a result of reading or listening to a news story (either positive ones like pride or negative feelings like anger) may inhibit our ability to evaluate the credibility of the story and the data that’s presented. Pausing to reflect on if and why you’re reacting strongly to a piece may allow you to step back and critique the news and/or data separate from your emotions.
4. Be thoughtful in the sharing of the information.
Before you share news articles or the data you’ve seen, pause and reflect on whether you’ve taken the time to fact-check the information to the best of your ability. If you stand by the conclusions drawn, consider what your intentions are for sharing the information. This can include, for example, trying to educate, attempting to prove a point, or creating an emotional reaction in others. Discerning your motivation for sharing the information may lead you to a different decision or at least prepare you for challenging encounters with family or friends.
Another helpful resource for evaluating the news: https://guides.library.cornell.edu/evaluate_news
Fletcher, R., & Nielsen, R. K. (2019). Generalised scepticism: How people navigate news on social media. Information, Communication & Society, 22(12), 1751–1769. https://doi.org/10.1080/1369118X.2018.1450887
Kahneman, D. (2003). A perspective on judgment and choice: Mapping bounded rationality. American Psychologist, 58(9), 697–720. https://doi.org/10.1037/0003-066X.58.9.697
Weber, E. U., & Johnson, E. J. (2009). Mindful Judgment and Decision Making. Annual Review of Psychology, 60(1), 53–85. https://doi.org/10.1146/annurev.psych.60.110707.163633