Variational sparse Gaussian-process method for detecting spatially variable genes and cellular interactions in spatial transcriptomics

Advanced spatially resolved transcriptomic (SRT) technologies preserve the spatial context of gene expression within tissues, enabling the study of context-dependent transcriptional regulation. Here, we propose a variational inference-assisted sparse Gaussian-process (VISGP) framework for identifying spatially variable genes (SVGs) and inferring spatially dependent cellular interactions from SRT data. VISGP combines sparse Gaussian-process approximations with variational inference via inducing variables to reduce computational and memory costs while enabling gene-specific adaptation of spatial covariance structures. Across simulated data and four real SRT datasets, VISGP detected more SVGs than existing methods and identified 85 spatially constrained ligand–receptor pairs that were missed by alternative approaches. Together, VISGP provides a scalable and statistically grounded strategy for decoding spatial gene regulation and cell–cell communication, yielding biological insights into cellular heterogeneity and cancer pathology.