UE-CLIP: Entropy-Guided Adaptive Prompt Sparsification for Efficient Open-Vocabulary Semantic Segmentation
编号:183
访问权限:仅限参会人
更新:2026-11-15 17:48:35
浏览:17次
In-person
摘要
Open vocabulary semantic segmentation applies the dense prediction paradigm to the open-vocabulary setting using the knowledge about the categories contained in the visionlanguage models like the CLIP model. However, in most existing segmentation methods based on CLIP, the computational cost of prompt generation remains constant for each individual image but might vary across different categories out of which only some are relevant for that specific image. In this paper, we study the task of prompt selection as a form of dynamic allocation of resources and introduce our proposed method, Uncertainty-Entropy Guided CLIP (UE-CLIP). UE-CLIP uses the pre-trained CLIP ViT-L/14-336 image encoder and a class presence head. Prompt selection for the cross-attention is guided by binary Shannon entropy and a threshold that can be learned, hence selecting the most relevant and confident prompts. The selected prompts yield confidence-weighted scene tokens that get aggregated along with the visual feature after refinement through ViT-Adapter and decoding through UperNet. On the ADE20K dataset, UE-CLIP attains CLIP-SP mIoU level (51.89%) and demonstrates 76.7% reduction of the number of active prompts from 30 to 7.0 and 0.23× cost compared to 1.00×.
稿件作者
Ameena Naik
SGBIT
Salika Khatib
SGBIT
Hemant Darur
Kle University
发表评论