A spatial augmented reality system to assist in template-based manual drilling of aircraft panels

Aircraft manufacturing involves manually drilling numerous high-precision holes by experienced workers using drilling templates. Most of the previous AR-assisted drilling research is related to drilling in surgery and mining, which involve workflows distinct from those in aircraft manufacturing. Some spatial augmented reality (SAR) assistance systems have been used to project virtual images onto the aircraft panels showing hole positions to assist in manual drilling, as they do not require additional wearable or handheld devices, support hands-free operation, and are more suitable for long-duration work compared to other types of augmented reality assistance systems. However, existing systems only display hole positions, lacking guidance for some other important steps in template-based manual drilling. This work presents a SAR-assisted system for template-based manual drilling of aircraft panels that guides a worker through the entire workflow of template-based manual drilling. It consists of a smart shelf to assist in retrieving and returning drilling templates and checking for errors, a binocular vision system to automatically determine drilling template installation positions by detecting and matching location holes, and a projector to display SAR drilling instructions. A user study with 16 participants was conducted to evaluate the system usability. The results showed that compared to traditional documentation guidance, the developed SAR system can speed up manual drilling and provide better perceived ease-of-use and lower perceived workload. The limitations of this research are also discussed, and some directions for future work are provided.