Please use this identifier to cite or link to this item:
http://hdl.handle.net/10397/76200
Title: | Regularizing neural networks via retaining confident connections | Authors: | Zhang, SG Hou, YX Wang, BY Song, DW |
Issue Date: | 2017 | Source: | Entropy, July 2017, v. 19, no. 7, 313, p. 1-14 | Abstract: | Regularization of neural networks can alleviate overfitting in the training phase. Current regularizationmethods, such as Dropout and DropConnect, randomly drop neural nodes or connections based on a uniform prior. Such a data-independent strategy does not take into consideration of the quality of individual unit or connection. In this paper, we aim to develop a data-dependent approach to regularizing neural network in the framework of Information Geometry. A measurement for the quality of connections is proposed, namely confidence. Specifically, the confidence of a connection is derived from its contribution to the Fisher information distance. The network is adjusted by retaining the confident connections and discarding the less confident ones. The adjusted network, named as ConfNet, would carry the majority of variations in the sample data. The relationships among confidence estimation, Maximum Likelihood Estimation and classical model selection criteria (like Akaike information criterion) is investigated and discussed theoretically. Furthermore, a Stochastic ConfNet is designed by adding a self-adaptive probabilistic sampling strategy. The proposed data-dependent regularization methods achieve promising experimental results on three data collections including MNIST, CIFAR-10 and CIFAR-100. | Keywords: | Information geometry Neural networks Regularization Fisher information |
Publisher: | Molecular Diversity Preservation International (MDPI) | Journal: | Entropy | ISSN: | 1099-4300 | EISSN: | 1099-4300 | DOI: | 10.3390/e19070313 | Rights: | © 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/). The following publication Zhang, S. G., Hou, Y. X., Wang, B. Y., & Song, D. W. (2017). Regularizing neural networks via retaining confident connections. Entropy, 19(7), (Suppl. ), 313, - is available athttps://dx.doi.org/10.3390/e19070313 |
Appears in Collections: | Journal/Magazine Article |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
Zhang_Regularizing_Neural_Networks.pdf | 308.11 kB | Adobe PDF | View/Open |
Page views
111
Last Week
2
2
Last month
Citations as of Apr 14, 2024
Downloads
55
Citations as of Apr 14, 2024
SCOPUSTM
Citations
5
Last Week
0
0
Last month
Citations as of Apr 19, 2024
WEB OF SCIENCETM
Citations
4
Last Week
0
0
Last month
Citations as of Apr 18, 2024
Google ScholarTM
Check
Altmetric
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.