Structure preservation in neural networks : approximation properties and partial differential equation solvers

Abstract

This dissertation focuses on incorporating structure-preserving concepts into the design and implementation of neural networks (NNs). As a versatile and quintessential model of deep learning, NNs have been widely applied in various scientific and engineering fields. However, practical applications often rise to the challenge of ensuring the physical or mathematical properties of the underlying problem, leading to external constraints on the network output or training process. Therefore, it is crucial to develop a systematic NN design that can preserve the intrinsic structure while maintaining the flexibility and expressiveness of deep learning. To address this challenge, we draw inspiration from the structure-preserving concept in numerical analysis, which aims to preserve the intrinsic structures of physical systems, such as conservation laws or symmetries. We incorporate this concept into the design and implementation of NNs, and then investigate the challenges and opportunities that arise from this approach. The structure-preserving properties arose from the practical applications often impose additional challenges, thereby we need to investigate whether NNs can still maintain effectiveness and efficiency under these constraints. Additionally, we explore the implications of exploiting these structure-preserving properties as an inductive bias, which can help to improve the performance and physical fidelity of NN-based models. Following this discussion, we propose two main research directions in this dissertation. We first consider the approximation properties under a highly constrained training process, which has unique advantages in practical applications. Recent experimental research proposed a novel permutation-based training method, which exhibited a desired classification performance without modifying the exact weight values. In the first part of this dissertation, we provide a theoretical guarantee of this permutation training method by proving its ability to guide a shallow network to approximate any one-dimensional continuous function. Our numerical results further validate this method's efficiency in regression tasks under various initializations. The notable observations during weight permutation suggest that permutation training can provide an innovative tool for describing network learning behavior. In the second part of this dissertation, we consider the application of solving partial differential equations (PDEs) with NNs, which has shown great potential in various scientific and engineering fields. However, most existing NN solvers mainly focus on satisfying the given PDEs, without explicitly considering intrinsic physical properties, such as mass conservation or energy dissipation. This limitation can result in unstable or nonphysical solutions, particularly in long-term simulations. To address this issue, we propose Sidecar, a novel framework that enhances the physical consistency of existing NN solvers for time-dependent PDEs. Inspired by a traditional structure-preserving numerical approach, our Sidecar framework introduces a small network as a copilot, guiding the primary function-learning NN solver to respect the structure-preserving properties. This framework is designed to be highly flexible, enabling the incorporation of structure-preserving principles from diverse PDEs into a wide range of NN solvers. Our experimental results on benchmark PDEs demonstrate improvements in the accuracy and physical consistency of existing NN solvers.