CompRess: Self-Supervised Learning by Compressing Representations

October 28, 2020 · Entered Twilight · 🏛 Neural Information Processing Systems

"Last commit was 5.0 years ago (≥5 year threshold)"

Evidence collected by the PWNC Scanner

Repo contents: LICENSE, Readme.md, cache_feats.py, docs, eval_cluster_alignment.py, eval_knn.py, eval_linear.py, kmeans.py, models, nn, tools.py, train_kmeans.py, train_student.py, train_student_one_queue.py, train_student_without_momentum.py, util.py

Authors Soroush Abbasi Koohpayegani, Ajinkya Tejankar, Hamed Pirsiavash arXiv ID 2010.14713 Category cs.CV: Computer Vision Cross-listed cs.LG Citations 99 Venue Neural Information Processing Systems Repository https://github.com/UMBCvision/CompRess ⭐ 80 Last Checked 1 month ago

Abstract

Self-supervised learning aims to learn good representations with unlabeled data. Recent works have shown that larger models benefit more from self-supervised learning than smaller models. As a result, the gap between supervised and self-supervised learning has been greatly reduced for larger models. In this work, instead of designing a new pseudo task for self-supervised learning, we develop a model compression method to compress an already learned, deep self-supervised model (teacher) to a smaller one (student). We train the student model so that it mimics the relative similarity between the data points in the teacher's embedding space. For AlexNet, our method outperforms all previous methods including the fully supervised model on ImageNet linear evaluation (59.0% compared to 56.5%) and on nearest neighbor evaluation (50.7% compared to 41.4%). To the best of our knowledge, this is the first time a self-supervised AlexNet has outperformed supervised one on ImageNet classification. Our code is available here: https://github.com/UMBCvision/CompRess