Towards a Universal Gating Network for Mixtures of Experts

November 03, 2020 · Entered Twilight · 🏛 arXiv.org

"Last commit was 5.0 years ago (≥5 year threshold)"

Evidence collected by the PWNC Scanner

Repo contents: .gitignore, README.md, archs, augments.py, config.py, merge.py, mnist, mnist_cifar10, requirements.txt, run_experiments.bat, run_train_pan.bat, run_train_source_net.bat, setup.cfg, train_gan.py, train_pan.py, train_source_networks.py, utils.py

Authors Chen Wen Kang, Chua Meng Hong, Tomas Maul arXiv ID 2011.01613 Category cs.LG: Machine Learning Cross-listed cs.NE Citations 2 Venue arXiv.org Repository https://github.com/cwkang1998/network-merging ⭐ 3 Last Checked 2 months ago

Abstract

The combination and aggregation of knowledge from multiple neural networks can be commonly seen in the form of mixtures of experts. However, such combinations are usually done using networks trained on the same tasks, with little mention of the combination of heterogeneous pre-trained networks, especially in the data-free regime. This paper proposes multiple data-free methods for the combination of heterogeneous neural networks, ranging from the utilization of simple output logit statistics, to training specialized gating networks. The gating networks decide whether specific inputs belong to specific networks based on the nature of the expert activations generated. The experiments revealed that the gating networks, including the universal gating approach, constituted the most accurate approach, and therefore represent a pragmatic step towards applications with heterogeneous mixtures of experts in a data-free regime. The code for this project is hosted on github at https://github.com/cwkang1998/network-merging.