Attention Heatmap Drift in a Contrastively Pretrained Vision–Language Model: A Controlled Matched-Learning-Rate Comparison of Full Fine-Tuning and Low-Rank Adaptation

Mon, 06 Apr 2026 00:00:00 +0000

Authors: Ruize Xia
DOI:
Source: ORCID record

Abstract

Fine-tuning contrastive vision-language models such as CLIP is an essential step for downstream task specialization, yet the associated changes in internal attention geometry remain insufficiently characterized. This preprint investigates attention heatmap drift in a contrastively pretrained vision-language model through a controlled matched-learning-rate comparison of full fine-tuning and low-rank adaptation. The study uses a multi-faceted metric suite, including CLS-to-patch attention entropy, effective receptive field, concentration, head diversity, attention rollout, representational analysis, and subset-sensitivity tests. The results provide a structural lens for understanding transformer adaptation and suggest that preserving attention structure is an important mechanism for maintaining foundation-model generalization.

Preprint | Artificial Minds, Human Values

Attention Heatmap Drift in a Contrastively Pretrained Vision–Language Model: A Controlled Matched-Learning-Rate Comparison of Full Fine-Tuning and Low-Rank Adaptation

Abstract

Read the paper