Multi-output shrunken regression trees

Abstract

The analysis of the increasingly complex and interdependent variables used in sectors such as supply chain management, healthcare, and finance requires multi-output regressions using advanced machine learning techniques. Drawing inspiration from Stein's paradox, this study explores the potential of using shrunken estimators to enhance the predictive performance of multi-output regression trees. Stein's paradox suggests that incorporating information from multiple, even unrelated distributions can improve the estimation of multiple means. Our approach diverges from the traditional practice of independently averaging values for each output by integrating closed-form shrunken estimators into each leaf of a multi-output regression tree. The theoretical contributions of our work are twofold: first, we formulate an optimization problem that balances prediction errors with a multi-output regularizer to derive the shrunken estimators; second, we validate the superiority of shrunken estimators over traditional sample means. Our computational experiments on both real-world and synthetic datasets show that our proposed multi-output shrunken regression trees outperform traditional methods, leading to significant improvements in prediction accuracy. Our novel approach to multi-output regression not only provides theoretical insights but also has practical benefits for diverse sectors.