Just the facts, ma'am: a guide to The Conversation's FactCheck process

There’s now a vast network of factcheck units around the world, operating in myriad different languages. However, none have a process quite like ours at The Conversation.

I’m The Conversation’s FactCheck Editor, so of course I’d say that. But don’t take my word for it. Ask Alexios Mantzarlis, chief of the International Fact-Checking Network at the prestigious Poynter Institute in the US, who has studied most of the factcheck units operating across the globe. His take on our process?

“The Conversation’s approach is a unique and fascinating model, and one that fact-checkers around the world could benefit from observing,” he said, noting that our blind review process was what made us different.

Alexios asked me to describe The Conversation’s fact checking process in detail for an article he was writing. You can read his write-up here and Twitter discussion about it here. I’ve reproduced below what I told him about our process (with a few edits for clarity).

I hope you enjoy reading our FactChecks as much as I do editing them.

The Conversation has quite a lengthy and structured process. I’ll break it down here, step by step.

Find the claim

It must be a quote from a politician, or other influential person in society. We don’t fact check “ideas” or concepts, we fact check quotes from people. We first assess whether the claim is actually checkable (is there a data set or a body of research evidence against which this claim could be checked?) and also whether or not there’s sufficient public interest in it. It’s no use trying to commission a FactCheck on a quote like “Australia has the best maternity leave policy in the OECD”. Phrases like “the best” are not checkable, they’re too subjective. It’s also better to try to avoid, where possible, attempting to fact check the future.

Find and commission the academic author

Once you have a claim worth checking, find an academic with the right expertise to research and write it up. This often involves contacting a few academics to find one who can be dispassionate in their approach and chat about how they’d tackle it, how quickly they could do it and what data sets they’d use to test the assertion. Then we commission the story. Our briefs are often quite detailed, and ask the authors to source every assertion they make in their FactCheck and answer specific questions, and steer clear of opinion, commentary and speculation. The FactChecks are not about how things “should” be, they’re about how things are. They’re about testing statements against the evidence and conveying those findings in plain English to a general audience, with a nice short verdict at the end. The verdict doesn’t always have to be “right” or “wrong”. It could be something more nuanced than that. This is about educating the audience, not trying to win a “gotcha” moment. No jargon, no emotive language, and keep it tight – preferably around 800 words.

Seek sources and comment

Contact the person whose claim you’re fact checking, or their spokesperson. Best to do this by email so there’s an email trail showing you went to them for comment. Explain you’ve commissioned the FactCheck and ask them a) for a data source to support the assertion you’re fact checking and b) whether there’s any further comment they’d like us to include in the story. Often we cannot include their full answer in the main article because of word length limits but we can include part of it and then tell the reader to click on a link to read the answer in full. Allow the person you’re fact checking sufficient time to reply.

Find the blind reviewer

While the academic is working away on the FactCheck, find a second academic with similar expertise to be the blind reviewer. It’s very important the blind reviewer does not know the identity of the author of the main article. This often means not disclosing the gender of the main author and trying where possible to find a blind reviewer academic from a different university or state from the main author. I’ll always impress upon them the importance of them staying naive to the name of the main author. This blind reviewer also needs to be dispassionate, not activist, in their approach to writing the article.

Edit with a critical eye

When the main author academic files his or her piece, I will edit it. That usually means putting a nice lead on it, editing out any opinion or jargon that’s crept in. I also put on my “Devil’s Advocate” hat and read the piece with a very critical eye and try to find holes in the author’s argument or weaknesses in their own sourcing. Once I have a version I am happy with, I send it back to the academic to check and hopefully approve. If they want further changes, we workshop those together.

At this point, I’ll often ask the opinion of a senior colleague. I’ll ask them to read over what we have got so far and see if they can identify weaknesses or suggest changes for clarity. This is a valuable step.

Get it blind reviewed

Once it’s looking good and the academic has approved it, I’ll take all the names off it and send it to the blind reviewer. They read over the anonymous draft and check the author “got it right”. Have they sourced their assertions properly? Did they cherry-pick the data? Did they accurately convey the evidence-based consensus among experts or have they gone out into fringe territory? Is there a way to communicate these ideas more clearly? If the reviewer sees room for improvement, they convey that to me and I workshop those improvements with the original academic author. If the blind reviewer is happy with the FactCheck, they write a short one-par review at the end saying something like “This is a sound analysis” and adding any further points they feel are needed.

Get a fresh pair of eyes on it

Then I ask a senior colleague to copytaste it. That means edit it for errors, look at it with fresh eyes to see if any of it is speculation passed off as fact, find holes in the argument, look for potential legal problems. If they see problems, we go back to the first author and get them to fix it, and then run those fixes past the blind reviewer, then it goes back to the copytaster.

If the copytaster finds no problems, we ask the author and the blind reviewer to have one final, forensic read-through of the whole thing and stress that this is their very last chance to fix any potential problems before publication. If there’s anything they’re not rock-solid, stand-up-in-court-and-say-it certain about, I ask them to delete it: when in doubt, take it out.

Publish

After the author and the blind reviewer have approved the final version, we publish and distribute on social media.

And if something goes wrong?

If we become aware that a correction is needed, we consult first with the author of the article to find the most accurate wording. We correct the article and note at the foot of the article it has been corrected, and how. Corrections are very rare, but when they do occur the response from readers is generally very positive – readers appreciate an honest and open corrections policy.

What do you think? Please share your thoughts and feedback in the comments section below.

Have you ever seen a “fact” worth checking? The Conversation’s FactCheck asks academic experts to test claims and see how true they are. We then ask a second academic to review an anonymous copy of the article. You can request a check at checkit@theconversation.edu.au. Please include the statement you would like us to check, the date it was made, and a link if possible.