Regression learning with continuous and discrete data

Abstract

Machine learning has achieved enormous successes in many different application areas of data mining in the last twenty years. Regression is a big branch of learning problems. This thesis investigates several topics in regression learning problems from the perspective of learning theory and asymptotic theory. First, we study a pairwise regularized least squares learning algorithm using the Kronecker product kernels. This pairwise learning model covers both score-based ranking problems and non-linear metric learning problems. A rank-independent non-asymptotic convergence rate of the obtained pairwise learning algorithm is derived. The pairwise learning algorithm achieves the minimax optimal learning rate, which is also derived in this thesis. Second, we propose an empirical feature-based sparse approximation algorithm for privacy consideration. Instead of using sensitive private data, empirical features are computed with published unlabeled data (without privacy issues). Summary statistics instead of raw data are used to protect private information. This semi-supervised learning algorithm achieves both sparsity and approximation accuracy. Third, we study the asymptotic theory of a modified Poisson estimator for discrete grouped and right-censored (GRC) count data. Asymptotic theoretical properties are derived under milder conditions on the information matrix of observations and results apply to both stochastic and fixed regressors. Results in this thesis improve existing results on modified Poisson estimators for GRC counts, where stochastic regressors with strictly positive definite Fisher information matrices are studied, significantly. The big data performance of this estimator is investigated with data on drug use in America.