Misleading statistics and data: how to protect yourself against bad statistics

"74% of firms say they want to be 'data-driven,' but only 29% say they are good at connecting analytics to action." — Forrester

Calling yourself data-driven means nothing if the data is wrong.

Credible numbers eliminate the guesswork. They combine experience and intuition with concrete evidence to produce decisions that actually move the needle. But intuitive dashboards, charts, and graphs can mask a troubling reality: misleading statistics and data.

It's easy to get blinded by the absoluteness of numbers, especially when they support a conclusion you already wanted. Failing to recognize false statistics is a direct threat to good decision-making. It encourages you to push the wrong buttons with full confidence, and that's where the real danger lies.

Here's how to spot misleading statistics, understand the common ways they distort reality, and know when data is solid enough to act on.

What is a misleading statistic?

A misleading statistic is a data point, figure, or visual representation that is inaccurate, false, or manipulated to convey a distorted or biased message. Misleading statistics arise from errors or biases in how data is collected, organized, or presented.

They lead to incorrect conclusions, poor decisions, and false confidence. The most common culprits: selective bias, neglected sample size, faulty correlations, and manipulative visuals.

Misleading statistics appear when a fault, deliberate or not, enters one of three key stages:

• Collecting: Using small sample sizes that produce big-sounding numbers with little statistical significance.
• Organizing: Omitting findings that contradict the point the researcher is trying to prove.
• Presenting: Manipulating visual or numerical data to influence perception.

Bad statistics appear in news outlets, ad campaigns, and even scientific literature. A striking 33.7% of scientists have admitted to misusing statistics to support their research. Even trusted gatekeepers of information are not immune.

Selective bias and false statistics

A study by Elizabeth Loftus tested how language shapes eyewitness testimony. Subjects watched a film of multiple car accidents, then were asked: "About how fast were the cars going when they smashed into each other?"

Other subjects were asked the same question with the word "smashed" replaced by softer verbs:

• Contacted
• Hit
• Bumped
• Collided

The stronger the verb, the higher the speed estimate. Subjects shown the stronger verb were also more likely to report seeing broken glass, even though no broken glass appeared in the film.

Source

Language is just one lever for selection bias. The Advertising Standards Authority (ASA) forced Colgate to drop its claim that "over 80% of dentists recommend Colgate" because the claim implied those dentists preferred Colgate over competing brands. The actual survey question asked whether dentists would recommend using any toothpaste over brushing alone. By cherry-picking the response, Colgate created a very different impression than the data supported.

Selective bias occurs when samples or data are incomplete or cherry-picked to skew perception. The framing of a question, the choice of who gets surveyed, and what gets reported all shape what the numbers appear to say.

Neglected sample size and false precision

(source)

90 out of 100 people answering "yes" is 90%. So is 900 out of 1,000. The percentages match, but the statistical weight is very different. Smaller sample sizes almost guarantee dramatic-looking results.

Never accept percentages at face value. In the words of biochemistry researcher Ana-maria Sundic:

"To ensure that the sample is representative of a population, sampling should be random, i.e. every subject needs to have equal probability to be included in the study. It should be noted that sampling bias can also occur if the sample is too small to represent the target population."

Extreme results warrant extra scrutiny. The more surprising the finding, the more important it is to check the sample behind it.

Faulty correlations and causation

"Correlation doesn't mean causation."

You've heard it before, and it keeps being true. When two variables correlate, the actual explanation is usually one of four things:

• Y causes X.
• X causes Y.
• A third factor triggers both X and Y.
• The correlation is pure chance.

Researchers and the people who consume their work fall into trouble through number fetishism and correlation hunting. Tyler Vigen compiled funny misleading statistics examples that make this point vividly.

This graph shows a striking correlation between the number of people who drowned falling into a pool and the number of films Nicolas Cage appeared in:

And this one links deaths by bedsheet entanglement to per-capita cheese consumption:

Cutting cheese consumption and Nicolas Cage's filmography will not save lives. But the graphs look convincing.

When researchers face pressure to produce useful findings or validate a hypothesis, the temptation to declare a premature "eureka" is real. Throw enough variables at each other and a correlation will appear. That's not insight; it's noise wearing a graph.

Misleading graphs and visuals

Data visualizations turn raw numbers into pictures of relationships, trends, and patterns. They can clarify complex information fast, and they can distort it just as quickly.

Data journalist Alberto Cairo, in his book Graphics, Lies, Misleading Visuals, documents misleading statistics examples from marketing ads, political campaigns, and news coverage.

One well-known case is the Terri Schiavo case, a right-to-die legal dispute in the US. A graph used during coverage showed how different political groups felt about the removal of life support:

At a glance, the graph suggests Democrats were three times more likely than Republicans or Independents to support the court's decision. A closer look reveals a 14% difference. The Y-axis starts at 50 instead of 0, making a modest gap look enormous.

When reviewing graphs and visuals, watch for:

• Truncated axes: A Y-axis that doesn't start at zero exaggerates differences.
• Uneven intervals and odd scales: Inconsistent increments distort trends.
• Missing context: One graph in isolation tells half a story. Compare similar data across multiple sources before drawing conclusions.

Safeguarding against misleading data and statistics

Bad statistics create shocking headlines, inflate conversion claims, and steer decisions in the wrong direction, sometimes knowingly, sometimes not. The defence is a habit of healthy scepticism.

When you encounter convincing data, run through these questions before acting on it:

Who is doing the research? Research is expensive. Check who is sponsoring it, what they stand to gain from the results, and whether they have a product or agenda tied to the outcome. An independent university study carries different weight than a report commissioned by the company whose product it evaluates.

Can the sample size and study length bear the weight of the claim? Inspect the numbers behind the headline. Small samples and short timeframes produce dramatic-looking results that rarely hold up at scale.

Are the visuals represented fairly? Check that scales start at zero, intervals are even, and the chart type suits the data. A statistic that needs a manipulated graph to land its point is a statistic worth questioning.

Is the research framed honestly? Review the language used, how the question was worded, and who was surveyed. Loaded language and narrow samples produce answers that confirm what the researcher wanted to find.

Greet new information with curiosity and scepticism. The numbers on your dashboards, reports, and analytics are only as trustworthy as the data behind them, and that's worth protecting.

Frequently asked questions about misleading statistics