DeepEBC : compressing the pre-trained LLMs with error-bounded lossy compression

Abstract

With the rapid evolution of LLMs, the size of the model has increased dramatically to achieve high accuracy, which brings strong memory and storage overhead for both LLM training, fine-tuning, and inference. Efficiently compressing the model parameter with a high compression ratio while guaranteeing the model accuracy becomes a challenging issue and is still largely under exploration. In this study, we employ the error-bounded lossy compressor, cuSZp, to compress model parameters. The algorithm achieves a compression ratio exceeding threefold while maintaining model inference accuracy loss within 0.5%. Furthermore, we have enhanced the algorithm to better accommodate the data characteristics of LLMs, since LLM’s model parameters may occasionally contain outliers. We evaluate each data block prior to compression; if the proportion of outliers within a block falls below a specified threshold, we apply specialized processing to that block to further improve compression efficiency. Additionally, we have extended the original algorithm to directly compress float16 tensors, offering greater convenience for models utilizing float16 parameters compared to existing quantization methods. Finally, we select distinct hyperparameter error bounds for each tensor based on the magnitude of the gradients and the variance of the model parameters, thereby further enhancing inference accuracy. Extensive experiment results based on the most popular benchmarks show that our proposed approach can achieve a compression ratio of up to 3.67 × while ensuring that the loss of inference accuracy is kept below 0.5%.