SciBERT: A Pretrained Language Model for Scientific Text

March 26, 2019 · Entered Twilight · 🏛 Conference on Empirical Methods in Natural Language Processing

"Last commit was 5.0 years ago (≥5 year threshold)"

Evidence collected by the PWNC Scanner

Repo contents: .dockerignore, .gitignore, Dockerfile, LICENSE.txt, README.md, allennlp_config, data, misc, requirements.txt, results, scibert, scripts, setup.py

Authors Iz Beltagy, Kyle Lo, Arman Cohan arXiv ID 1903.10676 Category cs.CL: Computation & Language Citations 3.5K Venue Conference on Empirical Methods in Natural Language Processing Repository https://github.com/allenai/scibert/ ⭐ 1669 Last Checked 1 month ago

Abstract

Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SciBERT, a pretrained language model based on BERT (Devlin et al., 2018) to address the lack of high-quality, large-scale labeled scientific data. SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. We evaluate on a suite of tasks including sequence tagging, sentence classification and dependency parsing, with datasets from a variety of scientific domains. We demonstrate statistically significant improvements over BERT and achieve new state-of-the-art results on several of these tasks. The code and pretrained models are available at https://github.com/allenai/scibert/.