Daily BulletinDaily Bulletin

The Conversation

  • Written by Adrian Barnett, Professor of Public Health, Queensland University of Technology

Statistics is a useful tool for understanding the patterns in the world around us. But our intuition often lets us down when it comes to interpreting those patterns. In this series we look at some of the common mistakes we make and how to avoid them when thinking about statistics, probability and risk.

Have you ever experienced the perfect evening out? The weather was great, you got the best table in the house at your favourite restaurant, the food was delicious and the wine superb, and the conversation was sparkling.

Then have you made the mistake of trying to repeat the experience and ended up disappointed? This is because your perfect evening was due to a series of chance events that all fell in your favour. A great experience is like tossing a coin and getting a long run of heads – unusual and difficult to repeat.

When you try to repeat a perfect experience, at least one thing is likely to be imperfect the second time around. The couple at the next table are loud and boorish, the waiter gets your order wrong, your jokes fall flat, and so on.

Happily, it works both ways. So if you’re forced to repeat a terrible experience, it’s likely that it won’t be so bad the second time around.

This phenomenon is called “regression to the mean” or “reversion to mediocrity”, which sums up how unusual events are likely to be followed by more typical ones.

The polymath Sir Francis Galton coined the term when he noticed that tall parents tended to have children shorter than them, whereas short parents often had children who were taller than themselves.

For a parent to be unusually tall, the genetic coin had to be “heads” many times in a row. Repeating that feat of chance for their children is not impossible, but it is unlikely.

image Outliers in a population, such as very short or tall parents, will tend to gravitate towards the mean, such as by having children that are closer to the average height in that population. CC BY

Is that a trend?

Regression to the mean is driven by chance, and so it occurs wherever chance occurs, which means it occurs almost everywhere. It is prevalent in sport and can explain the “manager of the month curse” in football. This award is usually won by managers who have had four or more wins in a row, often because of a combination of skill and luck. When the luck runs out, the “curse” strikes.

Regression to the mean will even occur in this article as unusually long sentences will tend to be followed by shorter ones. Check if you don’t believe me. It occurs in the published literature on regression to the mean, as years with many published papers on the phenomenon tend to be followed by years with fewer papers.

This article will itself cause some regression to the mean if it spikes interest in the Wikipedia page, but that interest will inevitably wane.

Regression to the mean is mostly harmless, but it becomes a problem when the change it creates is misinterpreted.

For example, imagine you ran a hospital and were told that hospital-acquired infections were five times higher than average last month. A colleague tells you they know the cause and it can solved by using more prophylactic antibiotics.

You agree and in the following month you’re told that prophylactic antibiotic use is through the roof and infections have come down. Your mind makes a causal connection and you’re now convinced of the need for widespread prophylactic antibiotics, a potentially dangerous connection given that the unusual infection rate could have been due to chance events.

Now your hospital budget will be tighter because of the costs of using more antibiotics, and you’re contributing to serious problem of antibiotic resistance.

image If you track the quality of your nights out, they might fluctuate up and down, but will still hover around the mean. CC BY

Making sham treatments look good

Regression to the mean is unwittingly exploited by quacks who often see patients when they are at their lowest. As many diseases have a natural ebb and flow, seeing patients when they are at their worst is the best time to exploit regression to the mean, because any treatment will appear to cause improvements in enough patients to make it look broadly effective.

Telling the difference between regression to the mean and a real change can be difficult. A chronically ill patient may have a very bad day, but is that the early warning of a downward trajectory or just a blip due to a random cluster of events, such as a bad meal, poor sleep, or an ill-judged sprint for the bus?

Gathering more data using watchful waiting can be useful, as once a clear pattern emerges in a patient’s well-being, it is less likely to be the random ups and downs of regression to the mean.

Regression to the mean is everywhere. Being aware of it might help you avoid overreacting to unusual events. However, if you didn’t find this article interesting or useful then why not read another one on The Conversation – the chances are you’ll enjoy it more.

Authors: Adrian Barnett, Professor of Public Health, Queensland University of Technology

Read more http://theconversation.com/regression-to-the-mean-or-why-perfection-rarely-lasts-74694

Big Tobacco's decisive defeat on plain packaging laws won't stop its war against public health


Students in China heed their government's warnings against studying in Australia – less than half plan to come back


We live in an age of 'fake news'. But Australian children are not learning enough about media literacy


The Conversation


Prime Minister Interview with Ben Fordham, 2GB

FORDHAM: Thank you very much for talking to us. I know it's a difficult day for all of those Qantas workers. Look, they want to know in the short term, are you going to extend JobKeeper?   PRI...

Scott Morrison - avatar Scott Morrison

Prime Minister Scott Morrison interview with Neil Mitchell

NEIL MITCHELL: Prime minister, good morning.    PRIME MINISTER: Good morning, how are you?   MICHELL: I’m okay, a bit to get to I apologise, we haven't spoken for a while and I want to get t...

Scott Morrison - avatar Scott Morrison

Prime Minister Interview with Ben Fordham

PRIME MINISTER: I've always found that this issue on funerals has been the hardest decision that was taken and the most heartbreaking and of all the letters and, you know, there's been over 100...

Scott Morrison - avatar Scott Morrison

Business News

SEO In A Time of COVID-19: A Life-Saver

The coronavirus pandemic has brought about a lot of uncertainty for everyone across the world. It has had one of the most devastating impacts on the day-to-day lives of many including business o...

a Guest Writer - avatar a Guest Writer

5 Ways Risk Management Software Can Help Your Business

No business is averse to risks. Nobody can predict the future or even plan what direction a business is going to take with 100% accuracy. For this reason, to avoid issues or minimise risks, some for...

News Company - avatar News Company

5 Ways To Deal With Unemployment and Get Back Into the Workforce

Being unemployed has a number of challenges and they’re not all financial. It can affect you psychologically and sometimes it can be difficult to dig your way out of a rut when you don’t have a job ...

News Company - avatar News Company

News Company Media Core

Content & Technology Connecting Global Audiences

More Information - Less Opinion