How to Spot Misleading Charts, a Checklist

“Distrust any publication that doesn’t clearly mention or link to the sources of the stories they publish.” –Alberto Cairo, data visualization expert and author of How Charts Lie

Whether you are reading a social post, news article or business report, it’s important to know and evaluate the source of the data and charts that you view. Investigate the source by asking questions that help you get to the heart of who, what, where, when, and why the chart was created. Ask questions such as:

• Where did the data come from?
• How was the data collected?
• Who made the presentation and who paid for the work to be done?
• How much data was collected? For example, if you are viewing survey data, how many people did they survey?

To help you question the data, be sure to study the metadata, which is the data about the data. Read any accompanying documentation to gather useful insights from the metadata, such as when, how, and why the data was collected. For example, many organizations that share data with the public will provide documentation describing characteristics, ownership, formatting, suggested use, and many other details about the data.

Another question to ask is “What summarizations were made to the data?” When viewing summary numbers, evaluate if the summary number is appropriate. For example, a common pitfall can happen when summarizing skewed data such as salary data. When using the average instead of the median, we may be misled into believing a higher value represents the typical or middle value for the data. The middle value is better represented as the median.

This histogram shows the distribution of base NBA salaries that shows the data grouped into ranges (or bins). The median value of $3.8 million is the better summary value for showing a ‘typical’ or middle value for NBA salaries. The average is much higher at $7.2 million.

To learn more about distributions, check out the Data Distributions badge on Trailhead.

Learning Chart design best practices can help you accurately interpret data. Read on for questions to ask and chart design best practices to consider while reviewing charts.

Is the chart the right type for the data being presented?

To understand the right chart type for the data being presented, we need to consider quantitative and qualitative variables.

Quantitative variables can be measured numerically. Examples of quantitative variables are number of items in a set or height in inches. When displayed on an axis, quantitative variables will have scales with even intervals (0, 5, 10, and so on).

Qualitative variables can’t be measured numerically. Examples of qualitative variables are favorite food, movie genre, or Sales region. When displaying qualitative variables, the axis will show categories instead of scales with even intervals.

To learn in more detail about types of variables, check out the Variables and Field Types module in Trailhead and Tableau’s help article on Data Structure.

There are different chart types appropriate for the type of data being presented. Some common chart types appropriate for qualitative variables include bar charts and pie charts. Some common chart types appropriate for quantitative variables include scatterplots, line charts, histograms, box and whisker plots, and bar charts.

Watch out for charts that use types of charts not meant for the type of data being presented. For example, you may encounter line charts, which are meant for showing quantitative data, used with qualitative data—as shown in the inappropriate line chart below. The line chart makes it appear that there are even intervals between categories. When we study the axis, we discover categories of sectors (Agriculture, Arts, Clothing, and so on) are numeric and do not have equal intervals and it would not make any sense to connect them with the line. A bar chart would be the better alternative.

Inappropriate line chart (left) using a qualitative axis and an appropriate alternative bar chart (right).

Are there any misleading design practices used in the chart that might deceive or distract you from correctly interpreting the data?

While creative and intriguing designs can be eye-catching, some can distract from properly communicating important insights in the data.

For example, while pictograms that use symbols and images to convey information may add visual appeal, when they don’t follow best practices, they may mislead viewers—particularly when used to display size differences. This is evident in the following pictogram below, where you’re misled into seeing a much larger difference between the two categories than what actually exists.

The underlying data is that A = 100 and B = 300. Looking at the chart, viewers are more likely to interpret the area between A and B in the chart rather than height as the difference between A and B. Using area in this pictogram misleads us into thinking there is a 9-fold difference instead of the correct 3-fold difference. This pictogram violates the Principle of Proportional Ink described by Carl Bergstrom and Javin West at the University of Washington: when a shaded region is used to represent a numerical value, the area of that shaded region should be directly proportional to the corresponding value.

Two pictogram charts: the pictogram on the left showcases two smiley faces, a smaller “A”, and larger “B”. The pictogram on the right showcases the same “A” and “B” but also includes how many smaller “A” smiley faces would make up the larger “B”. While value B is only 3 fold more than A, the larger smiley faces lead the reader to see a 9 fold difference instead.

Take the Guidelines to Recognize Misleading Charts Trailhead where we cover more design practices to watch for.

For most graphs or charts, axes (singular: axis) create the structure for how data is displayed in a chart usually with a horizontal or x-axis and a vertical or y-axis. When spotting misleading charts, it’s important to be able to discern if the axes are being used appropriately.

Does the chart use the appropriate scales and intervals?

As discussed above, having even intervals is important when checking that the right type of chart is being presented. Even when viewing the right chart type, it is still important to check the scales and intervals. A missing value can create an uneven interval and lead to misinterpretation.

Two line graphs showing the same data with different intervals on the axis. The graph on the left is missing 2016, 2017 and 2018. Viewing a graph that includes the missing years creates a very different interpretation.

Do the axes start at zero?

While there is leeway with line charts, see Misleading Axes on Graphs, bar charts must always begin with a zero baseline. In other words, the bottom of the bar is zero. When bar charts don’t start at zero, we can be led to believe that there is a much larger difference between categories than in the actual data.

Two bar charts showing the same data, the bar chart on the right has the x-axis start at zero while the bar chart on the left starts at a value higher than zero and does not indicate the value the x-axis starts at (probably about $390K). The bar chart on the left leads us to believe that there is a very large difference in sales between the two regions. When we include the zero baseline, we can quickly see the accurate difference between the two regions.

Are there multiple axes?

There are a few legitimate use cases for multiple axes. For example, it can be effective to show the same set of data points on two different scales using dual y-axes such as inches and centimeters or degrees Fahrenheit and degrees Celsius. However, there are many instances where multiple axes can be misleading.

The dual-axis chart below shows the world GDP and GDP for the Americas. We are led to believe that GDP for The Americas is the same as and more than World GDP! How can that be? Notice the scales of the y-axis. The Americas scale doesn’t match the World GDP scale.

(Left) Misleading Dual Axis line graph using two y-axes with different scales. (Right) Alternative to dual axis graph showing World and The Americas GDP with same scale and a single axis.

Alternatively, when we remake the graph to show the Word GDP and The Americas using the same scale on a single axis. We now can see the accurate difference in GDP between The Americas and the World. Data with the same units of measure are best shown using the same scale on a single axis.

When encountering dual axes with two different units of measurement, in most cases, two separate graphs are a better option than a dual-axis chart. In the below example from Jon Schwabish’s discussion of dual axis charts, we see how the scale on the second axis can be manipulated to show differing patterns with the same data. In this example, a more accurate way to present these data is one two separate graphs.

Two dual axis line charts showing the same data but using different scales. Manipulating the ranges on the second axis shows two different stories and can mislead. Images from Avoiding the Dual Axis Chart by Jon Schwabish.

Two separate graphs vertically aligned allows the reader to make accurate comparisons between Fatalities and Miles per capita. Image from Avoiding the Dual Axis Chart by Jon Schwabish.

Remember to always check the axes when you view graphs to ensure the right axes are used for the type of data being used and the appropriate scales and intervals are being used.

Even charts created with accurate source data and appropriate design choices are subject to misinterpretation. The key is to think critically and take the time to thoughtfully evaluate the interpretations of data that you see portrayed in charts.

What types of comparisons are made in the interpretation?

Data Visualization expert and author Kathy Rowell says that we should always ask “Compared to What?”, an “essential question for great data analysis and data visualizations.”

Charts help us make informed comparisons and answer the right questions that lead to good decisions. It’s important to consider the context of the chart and understand the comparisons made. Even if a chart’s data is trustworthy, you might jump to incorrect conclusions if you don’t pause to make sure the chart is addressing the right questions first.

Ask if the entire context is being presented. For example, in How Charts Lie: Getting Smarter about Visual Information, Alberto Cairo discusses the following bar chart. The suggested takeaway is that the unemployment rate is increasing.

Bar graph showing unemployment increasing from July to August of 2017. Image from How Charts Lie by Alberto Cairo.

However, if you look at these two data points in the context of the entire year or even across multiple years, a different pattern emerges. Although the unemployment rate did increase between July 2017 and August 2017, the rate is actually going down over time with fluctuations occurring between months. Including this context tells a very different story.

Unemployment rate shown over multiple years, showing a downward overall trend from 2009 to 2018. Image from How Charts Lie by Alberto Cairo.

Is the interpretation appropriate for the data analysis shown?

Sometimes an appropriate chart and analysis can be misinterpreted. For example, you may come across correlation being interpreted as causation.

Correlation only shows how strongly variables are related. It doesn’t explain the how or why.

For example, ice cream sales correlate with the number of sunglasses sold. Are people buying ice cream because they bought sunglasses, or vice versa? No. The cause of both purchases is clearly something else. In this case, the cause may be the hot weather.

Language Matters

Pay attention to words used to describe the chart. Do the title and subtitles properly describe the chart? Do labels use inclusive language that avoids stereotyping when describing data about people? Are the words emotionally laden in a way that may influence the perception of the data? Be aware of how language can influence our perception of the conclusions being presented. And finally, consider whether the visualization was created with an inclusive and equitable lens. Consider the possibility of bias in the analysis and presentation of the data.

To learn more about how to present data through a more diverse, equitable, and inclusive lens, check out the Equity and Inclusion Guidelines for Data Visualization Trailhead.