A multidimensional compliance monitoring and AR-guided method for error-prone assembly process of aviation angle pieces based on TCN-improved cross-modal action recognition and intelligent process perception

Whether aviation angle pieces in aviation equipment are correctly assembled has a significant impact on structural stability and flight safety. In actual production, the assembly process of aviation angle pieces suffers from issues such as prone to picking similar wrong pieces, non-standard manual operations, and low assembly efficiency, which severely restrict the production efficiency and service life of aviation equipment. Therefore, this paper proposes a multidimensional compliance monitoring and AR-guided method for error-prone assembly process of aviation angle pieces based on TCN-improved cross-modal action recognition and intelligent process perception, providing a digital and intelligent solution for process monitoring and guidance at assembly site of small parts in the aviation equipment manufacturing industry. In this paper, a Progressive Group-interactive Length-aware Spatiotemporal Graph Convolutional Network (PG-LSTGCN) is proposed. At the temporal convolution (TCN) level, it incorporates a multi-scale group-interactive convolution to capture features of both single joint and inter-joint coordinated movements and a new spatiotemporal decoupled attention mechanism to adaptively extract features from action samples of varying durations. This design addresses the limitations of conventional temporal convolutions in recognizing complex joint motion patterns and sensitivity of existing models to action samples’ lengths. Moreover, by fusing assembly action features from image and skeletal modalities, a cross-modal recognition method is proposed for similar assembly actions of angle pieces, which effectively avoids the recognition ambiguity caused by relying solely on the skeletal modality when distinguishing similar assembly actions. Furthermore, a new intelligent assembly process perception algorithm based on process graph retrieval and dynamic matching is proposed, which can automatically perceive the current assembly progress and perform compliance monitoring of assembly operations, thereby ensuring an accurate, standardized, and efficient assembly workflow for angle pieces. Finally, a human-region-object multi-dimensional assembly monitoring and AR-based intelligent guidance system is developed, enabling real-time monitoring of angle piece types, assembly regions, and assembly workflows, as well as intelligent recognition, automatic process progression, and real-time tracking of the assembly process. Experimental results show that the proposed PG-LSTGCN network effectively improves recognition accuracy on both public and private action datasets. The cross-modal similar assembly action recognition method achieves an accuracy of 95.84% on the similar angle piece assembly action dataset, outperforming the single skeletal modality by 3.96%. The AR-assisted assembly region monitoring and intelligent process perception method reduces the frequency of incorrect angle piece selection and omitted assembly steps by over 90%, decreases the occurrence of non-standard actions by up to 48.39%, and improves assembly efficiency by up to 10.49%, while simultaneously alleviating the cognitive load and mental demands of assembly operators.