New physics at the Large Hadron Collider? Scientists are excited, but it's too soon to be sure

Last week, physicists at the Large Hadron Collider in Switzerland announced they might have discovered a brand new force of nature. Or, to be precise, they unveiled “new results which, if confirmed, would suggest hints of a violation of the Standard Model of particle physics”.

What does that mean? And why are they making such a big deal of it, while at the same time stopping short of claiming a new discovery?

The answers lie in the way particle physicists think about evidence and results, and what it would mean to find “a violation of the Standard Model”.

So what?

The Standard Model, devised between the 1950s and 1970s, has been enormously successful at explaining the behaviour of subatomic particles and three of the four fundamental forces we know about. The physicists at CERN think they’ve found a situation that the Standard Model can’t explain: where the model predicts a particle called a beauty quark should decay into other particles called muons and electrons at about the same rate, it looks like it actually decays into electrons more often than muons.

Read more: Evidence of brand new physics at Cern? Why we're cautiously optimistic about our new findings

This is exciting, because we already know the Standard Model doesn’t tell the whole story about what’s happening in the universe. It’s very good at telling us about matter and energy. But it doesn’t provide an account of the so-called dark matter and dark energy scientists believe must exist to explain the large-scale behaviour of stars and galaxies.

The Standard Model is also tremendously difficult to reconcile with our best explanation of gravity, Einstein’s theory of general relativity. The Standard Model is at best a step along the road to a complete theory of everything.

To go beyond the Standard Model, we need new empirical data. What we really need is evidence showing some prediction of the Standard Model is wrong, but not a prediction so central to the theory that we need to rebuild from the ground up.

That’s why the decay of beauty quarks is so interesting. The unexpected behaviour points to an area where the theory could be modified without having to start from scratch.

Sigmas and p-values

The reason scientists are cautious about the result is because it’s what’s called a 3-sigma finding.

To explain, let’s imagine you’re looking for fairies in the bottom of your garden. You start by assuming there are no fairies – that’s called your null hypothesis.

You then gather some observations seeking to reject that hypothesis. After analysing your data, you find there is a 90% probability that if there were no fairies in the garden, you would make observations like the ones you in fact made.

Read more: Explainer: what are fundamental particles?

This gives you what’s called a p-value. A 90% probability of observing the data you in fact observed if your null hypothesis were true is the same as a p-value of 0.9.

Basically, you’ve discovered you don’t have a strong reason to reject the assumption that your garden is fairy-free. That’s not the same thing as discovering a reason to believe that your null hypothesis is true.

The p-value is the probability of the evidence, given your null hypothesis, which is distinct from the probability that the null hypothesis is true, given your evidence. (In case this seems odd, consider that the probability that someone is funny given that they’re dad is not the same as the probability that someone is your dad given that they’re funny).

Sigma values like the “3-sigma” result correspond to p-values. At the LHC, the null hypothesis is the claim that the Standard Model is correct, and the observations are of particle interactions.

A 3-sigma result means there is a roughly 1 in 1,000 probability that observations at least as extreme as those gathered would occur, given the Standard Model. That’s substantially better than your quest to find fairies and does seem to call the Standard Model into question.

Why so cautious?

Physicists don’t usually crack open the champagne until they have a 5-sigma result.

A 5-sigma result tells you there would be a chance of less than one in a million of your observation if the Standard Model were correct. That’s like wandering into your garden and chatting with a small being with wings: your “no fairies” hypothesis is starting to look quite shaky.

Why do physicists look for a 5-sigma event? There are several reasons. The first is historical: they’ve been stung before. In 2011 physicists claimed to have measured neutrinos travelling faster than the speed of light. This measurement exceeded 3-sigma, but it turned out to be due to a faulty cable.

Physicist Tommaso Dorigo has been keeping a diary of measured events that reached or surpassed 3-sigma significance. He notes 6 previous claims that were later withdrawn.

Another reason for caution is the problem of multiple comparisons. If you carry out enough tests, you are bound to see something odd.

Suppose you flip a coin 100 times and get 50 heads and 50 tails. Now suppose you repeat the experiment 100 times (flipping the coin 10,000 times altogether).

Read more: Explainer: how does an experiment at the Large Hadron Collider work?

In some versions of the experiment you might see 20 heads and 80 tails. In some you see 10 heads and 90 tails. Both distributions are unlikely, assuming the coin is fair.

Do you therefore have evidence the coin is unfair? It seems doubtful. Even a fair coin will yield lopsided results sometimes.

The LHC is like a coin-flipping machine. It is constantly conducting experiments. To correct for this, physicists demand the very high 5-sigma standard. A 3-sigma result is noteworthy, but not yet a “discovery”.

Finally, there’s the adage that extraordinary claims require extraordinary evidence. The Standard Model is extremely well confirmed. It will take an extremely striking observation (such as on observation of an event that would be very unlikely if the standard model were true) to reduce confidence in the model.

What’s next?

The LHC is an extraordinarily complex experiment, and there are a lot of things that can go wrong with it. That makes it difficult to control for systematic errors.

So even reaching the 5-sigma level in itself might not be enough to confirm a new discovery. Indeed, three of the six withdrawn results documented by Dorigo reached the even higher 6-sigma level.

To confirm a discovery, ideally the results need to be replicated using a different experimental set up (one that doesn’t risk also replicating the same errors), preferably more than once. That’s why the physicists at CERN are hoping their results will be replicated by the Belle experiment in Japan.

The announcement from CERN thus may seem a bit premature. But Dorigo’s diary provides reason to be optimistic. He points out that all of the withdrawn results from particle accelerator experiments reached levels of significance that are even numbers (4 or 6-sigma), whereas genuine discoveries reached levels that are odd numbers (3 or 5-sigma).

Dorigo suggests we should take observations with odd-numbered sigma values very seriously. He’s making a joke. But behind the joke is a sociological observation: physicists don’t tend to publish 3-sigma results unless they’re confident that they will lead to a discovery. The physicists at CERN clearly believe they’re onto something, and so should we.