Please use this identifier to cite or link to this item:
Title: A confident information first principle for parameter reduction and model selection of boltzmann machines
Authors: Zhao, XZ
Hou, YX
Song, DW
Li, WJ 
Keywords: Boltzmann machine (BM)
Fisher information
Information geometry (IG)
Parametric reduction
Issue Date: 2018
Publisher: Institute of Electrical and Electronics Engineers
Source: IEEE transactions on neural networks and learning systems, May 2018, v. 29, no. 5, p. 1608-1621 How to cite?
Journal: IEEE transactions on neural networks and learning systems 
Abstract: Typical dimensionality reduction (DR) methods are data-oriented, focusing on directly reducing the number of random variables (or features) while retaining the maximal variations in the high-dimensional data. Targeting unsupervised situations, this paper aims to address the problem from a novel perspective and considers model-oriented DR in parameter spaces of binary multivariate distributions. Specifically, we propose a general parameter reduction criterion, called confident-information-first (CIF) principle, to maximally preserve confident parameters and rule out less confident ones. Formally, the confidence of each parameter can be assessed by its contribution to the expected Fisher information distance within a geometric manifold over the neighborhood of the underlying real distribution. Then, we demonstrate two implementations of CIF in different scenarios. First, when there are no observed samples, we revisit the Boltzmann machines (BMs) from a model selection perspective and theoretically show that both the fully visible BM and the BM with hidden units can be derived from the general binary multivariate distribution using the CIF principle. This finding would help us uncover and formalize the essential parts of the target density that BM aims to capture and the nonessential parts that BM should discard. Second, when there exist observed samples, we apply CIF to the model selection for BM, which is in turn made adaptive to the observed samples. The sample specific CIF is a heuristic method to decide the priority order of parameters, which can improve the search efficiency without degrading the quality of model selection results as shown in a series of density estimation experiments.
ISSN: 2162-237X
EISSN: 2162-2388
DOI: 10.1109/TNNLS.2017.2664100
Appears in Collections:Journal/Magazine Article

View full-text via PolyU eLinks SFX Query
Show full item record


Citations as of Nov 15, 2018


Last Week
Last month
Citations as of Dec 9, 2018

Page view(s)

Citations as of Dec 3, 2018

Google ScholarTM



Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.