TraceCluster: A Lightweight and Adaptive Clustering-Based Subgraph Attention Network for APT Detection in Provenance Graphs

Provenance graph-based anomaly detection, particularly for Advanced Persistent Threat (APT) detection, addresses the issues of large-scale graphs and data imbalance. However, existing methods struggle with information loss, high computational complexity, and low detection accuracy. To address the above challenges, this paper proposes TraceCluster, a lightweight and adaptive clustering-based Subgraph Attention Network (SAN) for APT detection in provenance graph. TraceCluster mitigates the neighborhood explosion problem by clustering nodes to partition large-scale graphs, thus reducing reliance on the global graph while preserving local neighborhood information. Furthermore, the method dynamically models complex inter-node dependencies within subgraphs. It employs an attention mechanism to adaptively highlight the most relevant connections. This enhances node representations and improves overall feature extraction. This design substantially reduces memory consumption and avoids the high computational complexity of global graph processing. In addition, an adaptive category-weighting loss function assigns variable weights to different classes, improving the detection of rare and anomalous behaviors. Experimental results show that on the OpTC dataset, the currently faster method is 37-fold and 3-fold slower than our approach in terms of inference time respectively. Furthermore, in the nine real-world scenarios of four evaluated datasets, TraceCluster outperforms state-of-the-art (SOTA) approaches in terms of overall performance, especially in node-level APT detection tasks.