Bayesian Compression for Natural Language Processing

October 25, 2018 · Entered Twilight · 🏛 Conference on Empirical Methods in Natural Language Processing

"Last commit was 7.0 years ago (≥5 year threshold)"

Evidence collected by the PWNC Scanner

Repo contents: Code, Data, Experiments, LICENSE, Posters, README.md

Authors Nadezhda Chirkova, Ekaterina Lobacheva, Dmitry Vetrov arXiv ID 1810.10927 Category cs.CL: Computation & Language Cross-listed cs.LG, stat.ML Citations 15 Venue Conference on Empirical Methods in Natural Language Processing Repository https://github.com/tipt0p/SparseBayesianRNN ⭐ 16 Last Checked 1 month ago

Abstract

In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these parameters are often concentrated in the embedding layer, which size grows proportionally to the vocabulary length. We propose a Bayesian sparsification technique for RNNs which allows compressing the RNN dozens or hundreds of times without time-consuming hyperparameters tuning. We also generalize the model for vocabulary sparsification to filter out unnecessary words and compress the RNN even further. We show that the choice of the kept words is interpretable. Code is available on github: https://github.com/tipt0p/SparseBayesianRNN