Hierarchical Differentiable Clustering for Block-Sparse Cross-Attention in Multi-Modal Satellite Data Fusion

Abstract

Multi-modal satellite data fusion presents a critical challenge for climate change prediction and environmental monitoring, as conventional Transformer-based cross-attention mechanisms suffer from quadratic computational complexity when processing thousands of image patches from heterogeneous sensors, such as Sentinel-2 optical imagery, Sentinel-1 SAR, MODIS thermal data, and GPM precipitation products. We introduce a reformulated cross-attention system that integrates a differentiable hierarchical clustering module to replace the dense softmax operation with a learned block-sparse attention structure. The proposed method first generates token embeddings from all satellite modalities using a shared Vision Transformer backbone with geospatial positional encodings. It then performs a hierarchical soft clustering of these tokens over multiple levels, where a binary tree structure is learned recursively through affine transformations that ensure nested parent-child relationships. A temperature-annealed soft assignment mechanism yields leaf-level cluster memberships, which are subsequently converted into a block-sparse mask for the cross-attention computation. This mask restricts attention to tokens belonging to the same semantic cluster, thereby reducing the per-layer complexity from quadratic to approximately linear with respect to the number of clusters. The clustering module, the attention parameters, and the prediction head are trained jointly using a composite loss that combines the task objective with regularization terms for balanced cluster sizes and hierarchical consistency