Auditing MLaaS inference service quality without ground truth via mutual information

Abstract

Machine Learning as a Service (MLaaS) paradigm offers an appealing solution for clients that have limited computational resources. It allows entities to train models with collected dataset and powerful cloud resources, and to deploy these models for inference. However, MLaaS currently faces significant challenges in ensuring trustworthy inference and service quality. The clients cannot verify that the inference results returned by service provider (SP) are the model’s actual inference results. Moreover, even if clients manage to ensure that the results are obtained through model inference, they are unable to determine the model’s service quality without ground truth. To address these concerns, we introduce an innovative framework to audit inference quality and integrity in MLaaS through a novel deep neural network (DNN) inspection method. In specific, our approach represents the inherent behavior of the model by collecting its intermediate layer outputs and quantifying the mutual information (MI) values derived from them. By benchmarking the model during the training process, the SP can record the characteristics of the correct model and its corresponding service quality. After receiving the auditing request, the auditor can evaluate the quality of the service by estimating its accuracy via mutual information. Moreover, it can confirm the integrity of the returned results by inspecting the intermediate layer output. In addition, we thoroughly analyze our scheme for various potential adaptive attacks. Through empirical studies, we verify the correctness, effectiveness, and robustness of our scheme for trustworthy MLaaS inference service.