Please use this identifier to cite or link to this item: http://hdl.handle.net/10397/76200
Title: Regularizing neural networks via retaining confident connections
Authors: Zhang, SG
Hou, YX 
Wang, BY
Song, DW
Keywords: Information geometry
Neural networks
Regularization
Fisher information
Issue Date: 2017
Publisher: Molecular Diversity Preservation International (MDPI)
Source: Entropy, 2017, v. 19, no. 7, 313 How to cite?
Journal: Entropy 
Abstract: Regularization of neural networks can alleviate overfitting in the training phase. Current regularizationmethods, such as Dropout and DropConnect, randomly drop neural nodes or connections based on a uniform prior. Such a data-independent strategy does not take into consideration of the quality of individual unit or connection. In this paper, we aim to develop a data-dependent approach to regularizing neural network in the framework of Information Geometry. A measurement for the quality of connections is proposed, namely confidence. Specifically, the confidence of a connection is derived from its contribution to the Fisher information distance. The network is adjusted by retaining the confident connections and discarding the less confident ones. The adjusted network, named as ConfNet, would carry the majority of variations in the sample data. The relationships among confidence estimation, Maximum Likelihood Estimation and classical model selection criteria (like Akaike information criterion) is investigated and discussed theoretically. Furthermore, a Stochastic ConfNet is designed by adding a self-adaptive probabilistic sampling strategy. The proposed data-dependent regularization methods achieve promising experimental results on three data collections including MNIST, CIFAR-10 and CIFAR-100.
URI: http://hdl.handle.net/10397/76200
ISSN: 1099-4300
EISSN: 1099-4300
DOI: 10.3390/e19070313
Appears in Collections:Journal/Magazine Article

Access
View full-text via PolyU eLinks SFX Query
Show full item record

Google ScholarTM

Check

Altmetric


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.