Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers

November 15, 2023 · Entered Twilight · 🏛 Conference on Empirical Methods in Natural Language Processing

Repo contents: Factbench.jsonl, LICENSE, README.md, annotation_tool, factcheck-GPT-benchmark.jsonl, fig, reproduce_tutorial.ipynb, src, subtasks, trail.json

Authors Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, Preslav Nakov arXiv ID 2311.09000 Category cs.CL: Computation & Language Citations 72 Venue Conference on Empirical Methods in Natural Language Processing Repository https://github.com/yuxiaw/Factcheck-GPT ⭐ 112 Last Checked 1 month ago

Abstract

The increased use of large language models (LLMs) across a variety of real-world applications calls for mechanisms to verify the factual accuracy of their outputs. In this work, we present a holistic end-to-end solution for annotating the factuality of LLM-generated responses, which encompasses a multi-stage annotation scheme designed to yield detailed labels concerning the verifiability and factual inconsistencies found in LLM outputs. We further construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document, aiming to facilitate the evaluation of automatic fact-checking systems. Preliminary experiments show that FacTool, FactScore and Perplexity.ai are struggling to identify false claims, with the best F1=0.63 by this annotation solution based on GPT-4. Annotation tool, benchmark and code are available at https://github.com/yuxiaw/Factcheck-GPT.