Explainable Fact Checking: A Survey

This repository and the accompanying webpage contain resources for the paper "Explainable Fact Checking: A Survey". In the paper, we offer a critical review of the state-of-the-art in automated fact-checking with a particular focus on interpretability.

Reference

If you find our work useful, please cite the paper as formatted below.

  @inproceedings{kotonya-toni-2020-explainable-automated,
      title = "Explainable Automated Fact-Checking: A Survey",
      author = "Kotonya, Neema  and
        Toni, Francesca",
      booktitle = "Proceedings of the 28th International Conference on Computational Linguistics",
      month = dec,
      year = "2020",
      address = "Barcelona, Spain (Online)",
      publisher = "International Committee on Computational Linguistics",
      url = "https://www.aclweb.org/anthology/2020.coling-main.474",
      pages = "5430--5443"
  }

Here is an overview of papers mentioned in this work, and more recent papers which have been added.

Introduction
Task Formulations
Datasets
- Naturally occurring claims
  - From social media
  - From fact-checking and news websites
- Hand crafted claims
  - From Wikipedia
  - From scientific journals
Fact Checking Systems
- By dataset
- By method
Shared Tasks
Explainable Fact Checking
Surveys
Tutorials

Introduction

Fact checking is the process of establishing the veracity of claims i.e., to distinguish between false stories (e.g., misattributions, rumours, hoaxes) and facts.

Over the past few years the use of deep learning methods for fact checking and fake news detection have become a popular. Indeed, several exciting breakthroughs have occurred in automated fact checking thanks in large part due to new datasets (e.g., FEVER) and advances in machine learning for NLP. However there are still some limitations in this research area, the one we focus on in this work in our work is explanations for automated fact checking.

The pipeline commonly employed for automated fact-checking consists of four parts (subtasks). We propose that post-hoc explanations are an important and necessary extension of this pipeline.

For an overview of the data and results mentioned in our survey, please visit this webpage.

Task Formulations

Here we list papers which address varied tasks related to fact checking and fake news detection.

Check-worthy Claim Detection
- Toward Automated Fact-Checking: Detecting Check-worthy Factual Claims by ClaimBuster (Hassan et al., 2017). [paper] [bib]
Fauxtography and Multimodal Fake News Detection
- FauxBuster: A Content-free Fauxtography Detector Using Social Media Comments (Zhang et al., 2018). [paper] [bib] [slides]
- Fact-Checking Meets Fauxtography: Verifying Claims About Images (Zlatkova et al., 2019). [paper] [bib]
- Eann: Event adversarial neural networks for multi-modal fake news detection (Wang et al., 2018). [paper] [bib]
Identifying Previously Fact-Checked Claims
- That is a Known Lie: Detecting Previously Fact-Checked Claims (Shaar et al., 2020). [paper] [bib]
- Where Are the Facts? Searching for Fact-checked Information to Alleviate the Spread of Fake News (Vo and Lee, 2020). [paper] [bib]
Neural Fake News Detection
- Defending against neural fake news (Zellers et al., 2019). [paper] [bib]
- Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News (Tan et al., 2020). [paper] [bib]
Rumour Verification and Resolution
- SemEval-2019 Task 7: RumourEval, Determining Rumour Veracity and Support for Rumours (Gorrell et al., 2019). [paper] [bib]
- Can Rumour Stance Alone Predict Veracity? (Dungs et al., 2018). [paper] [bib]
- SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours (Derczynski et al., 2017). [paper] [bib]
Stylometric Analysis of News Articles
- A stylometric inquiry into hyperpartisan and fake news (Potthast et al., 2017) [paper] [bib] [video]
- Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking (Rashkin et al., 2017). [paper] [bib]
Table-based Fact Verification
- TabFact: A Large-scale Dataset for Table-based Fact Verification (Chen et al., 2020). [paper] [bib]
Multi-hop Fact Checking
Error Correction of Claims

Fact Checking Datasets

List of fact checking, rumour verification and fake news detection datasets:

Datasets of naturally occurring claims

Social media

Claims from social media platforms sources e.g., Twitter, Facebook.

r/Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection (Nakamura et al., 2020). [paper] [data] [bib]
SemEval-2019 Task 7: RumourEval, Determining Rumour Veracity and Support for Rumours (Gorrell et al., 2019). [paper] [data] [bib]
All-in-one: Multi-task Learning for Rumour Verification (Kochkina et al., 2018). [paper] [data]† [bib]
SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours (Derczynski et al., 2017). [paper] [data] [bib]
Hyperpartisan Facebook Pages Are Publishing False And Misleading Information At An Alarming Rate (Silverman et al., 2017). [article] [data]
Detect Rumors in Microblog Posts Using Propagation Structure via Kernel Learning (Ma et al., 2017). [paper] [data] [bib]
Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational Threads (Zubiaga et al., 2016). [paper] [data] [bib]
CREDBANK: A Large-Scale Social Media Corpus with Associated Credibility Annotations (Mitra and Gilbert, 2015). [paper] [data] [bib]

† This dataset is an extention of the PHEME rumour dataset.

Fact checking and news websites

Claims for news and fact-checking platforms e.g., Snopes, Politifact.

Explainable Automated Fact-Checking for Public Health Claims (Kotonya and Toni, 2020). [paper] [data] [bib]
STANDER: An Expert-Annotated Dataset for News Stance Detection and Evidence Retrieval (Conforti et al., 2020). [paper] [data] [bib]
FakeCovid-- A Multilingual Cross-domain Fact Check News Dataset for COVID-19 (Shahi and Nandini, 2020). [paper] [data] [bib]
MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims (Augenstein et al., 2019). [paper] [bib] [data]
A Richly Annotated Corpus for Different Tasks in Automated Fact-Checking (Hanselowski et al., 2019). [paper] [code] [data] [bib]
Integrating Stance Detection and Fact Checking in a Unified Corpus (Baly et al., 2018). [paper] [data] [bib]
FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media (Shu et al., 2018). [paper] [data]
“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection (Wang, 2017). [paper] [data] [bib]
Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking (Rashkin et al., 2017). [paper] [data] [bib]
The Fake News Challenge (Pomerleau and Rao, 2017) [data]

Hand crafted

This covers claims which are generated manually e.g. through re-writing statements.

Wikipedia

TabFact: A Large-scale Dataset for Table-based Fact Verification (Chen et al., 2020). [paper] [data] [bib]
FEVER: a Large-scale Dataset for Fact Extraction and VERification (Thorne et al., 2018). [paper] [data] [bib]
Automated Fact-Checking of Claims from Wikipedia (Sathe et al., 2020). [paper] [data] [bib]
Generating Fact Checking Briefs (Fan et al., 2020). [paper] [bib]

Scientific journals

Fact or Fiction: Verifying Scientific Claims (Wadden et al., EMNLP 2020). [paper] [data] [bib]

Fact Checking Systems

A list of fact-checking and fake news detection systems.

Systems by Dataset

LIAR

Where is your Evidence: Improving Fact-checking by Justification Modeling (Alhindi et al., 2018). [paper] [bib] [code]
Generating Fact Checking Explanations (Atanasova et al., 2020). [paper] [bib]

FEVER

FEVER 1.0 Baseline
Combining Fact Extraction and Verification with Neural Semantic Matching Networks (Nie et al., 2019). [paper] [bib] [code]
UCL Machine Reading Group: Four Factor Framework For Fact Finding (HexaF) (Yoneda et al., 2018). [paper] [bib] [code]
UKP-Athene: Multi-Sentence Textual Entailment for Claim Verification (Hanselowski et al., 2018). [paper] [bib] [code]
Team Papelo: Transformer Networks at FEVER (Malon, 2018). [paper] [bib] [code]
Team DOMLIN: Exploiting Evidence Enhancement for the FEVER Shared Task (Stammbach and Neumann, 2019). [paper] [bib] [code]
GEAR: Graph-based Evidence Aggregating and Reasoning for Fact Verification [paper] [bib] [code]
Fine-grained Fact Verification with Kernel Graph Attention Network [paper] [bib] [code]

MultiFC

Time-Aware Evidence Ranking for Fact-Checking (Allein et al., 2020). [paper] [bib]

Systems by Method

Support Vector Machines

Fake News or Truth? Using Satirical Cues to Detect Potentially Misleading News. [paper] [bib]

Convolutional Neural Networks

“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection (Wang, 2017). [paper] [bib]
FAKTA: An Automatic End-to-End Fact Checking System (Nadeeem et al., 2019). [paper] [bib]

Recurrent Neural Networks

CSI: A Hybrid Deep Model for Fake News Detection (Ruchansky et al., 2017). [paper]
DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning (Popat et al., 2018). [paper] [bib]
Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking (Rashkin et al., 2017). [paper] [bib]
Where is your Evidence: Improving Fact-checking by Justification Modeling (Alhindi et al., 2018). [paper] [bib]

Transformers and Attention Networks

Two Stage Transformer Model for COVID-19 Fake News Detection and Fact Checking (Vijjali et al., 2020). [paper] [bib]

Hybrid

GCAN: Graph-aware Co-Attention Networks for Explainable Fake News Detection on Social Media (Lu and Li, 2020). [paper] [bib]
DTCA: Decision Tree-based Co-Attention Networks for Explainable Claim Verification (Wu et al., 2020). [paper] [bib]
XFake: Explainable Fake News Detector with Visualizations (Yang et al., 2019). [paper] [bib]

Shared Tasks

📣
indicates the shared task is ongoing!

Statement Verification and Evidence Finding with Tables (SEM-TAB-FACT) [Wang et al., 2021] 📣 (Ends on Jan 29 2021)
SciFact Claim Verifiation [Wadden et al., 2020] 📣
Fakeddit Multimodal Fake News Detection Challenge [Nakamura et al., 2020] 📣 (Ends on Feb 16 2021)
SemEval-2019 Task 7: RumourEval, Determining Rumour Veracity and Support for Rumours [Gorrell et al., 2019]
SemEval-2019 Task 8: Fact Checking in Community Question Answering Forums [Mihaylova et al., 2019]
The Fake News Challenge (FNC-1) [Pomerleau and Rao, 2017]
- A Retrospective Analysis of the Fake News Challenge Stance-Detection Task [Hanselowski et al., 2018]
The Fact Extraction and VERification (FEVER) Shared Task [Thorne et al., 2018]
SemEval-2017 Task 8: RumourEval: Determining rumour veracity and support for rumours [Derczynski et al., 2017]