Model-agnostic personalized adaptation for segment anything model

Abstract

The Segment Anything Model (SAM) and its family of models have made significant strides in open-set, prompt-driven instance segmentation. However, some closed-source SAM family models often face ethical, copyright, or commercial restrictions, limiting their accessibility and further personalized adaptation. To overcome these limitations, we introduce MapSAM, a model-agnostic, personalized plugin for SAM family models. MapSAM features a lightweight threshold learner that enables nuanced post-hoc processing of confidence maps, leading to improved segmentation accuracy. By leveraging mask-focused learning, our approach determines pixel-wise and hardness-aware thresholds, allowing for more effective adaptation to diverse datasets. Furthermore, we critically examine the limitations of the commonly used Dice loss, which can overlook sample hardness when allocating penalties. We theoretically demonstrate that the Mean Squared Error (MSE) loss complements Dice loss by providing a stronger focus on sample hardness. Through extensive experiments on seven diverse datasets using multiple SAM family models, we validate the effectiveness of MapSAM in achieving superior segmentation results, particularly in challenging domains. Our findings open up new avenues for personalized, open-set instance segmentation across various application areas, leveraging any closed-source SAM family model. Code will be available at https://github.com/wjc2830/MapSAM.git.