In-context meta LoRA generation

Abstract

Low-rank Adaptation (LoRA) has demonstrated remarkable capabilities for task-specific fine-tuning. However, in scenarios involving multiple tasks, training a separate LoRA model for each task results in considerable inefficiency in terms of storage and inference. Moreover, existing parameter generation methods fail to capture the correlations among these tasks, making multitask LoRA parameter generation challenging. To address these limitations, we propose In-Context Meta LoRA (ICM-LoRA), a novel approach that efficiently achieves task-specific customization of large language models (LLMs). Specifically, we use training data from all tasks to train a tailored generator, Conditional Variational Autoencoder (CVAE). CVAE takes task descriptions as inputs and produces task-aware LoRA weights as outputs. These LoRA weights are then merged with LLMs to create task-specialized models without the need for additional fine-tuning. Furthermore, we utilize in-context meta-learning for knowledge enhancement and task mapping to capture the relationship between tasks and parameter distributions. Consequently, our method achieves more accurate LoRA parameter generation for diverse tasks using CVAE. ICM-LoRA enables more accurate LoRA parameter reconstruction than current parameter reconstruction methods and is useful for implementing task-specific enhancements to LoRA parameters. Simultaneously, our method occupies only 283MB, which is about 1% of the storage space required by the original LoRA. The code is available at https://github.com/YihuaJerry/ICM-LoRA.