Projected augmented reality assembly assistance system supporting multi-modal interaction

As an application paradigm of augmented reality in industry, the projected augmented reality (PAR) system is helpful in assembly, improving the ability of human–computer assembly cooperation. Precious research mainly focuses on visualization whereas the problem of how users naturally interact with the system has still not been discussed. In this paper, we developed a PAR system supporting multi-modal interaction (MMI), with which users could assemble and interact naturally. First, we tested the performance of MMI (speech, gesture, and finger click) against the baseline condition of single mode interaction. Meanwhile, we also compare the performance difference between gesture and finger click in bare-hand interaction. It is found that finger click shows a higher success rate, faster response time, lower false errors, less instruction memory, and better user experience than the gesture in bare-hand interaction. In specific, when the assembly must be completed but the hands are occupied, MMI can take advantage of user interaction modalities and reduce the interaction burden. The results prove that MMI shows better interactive naturalness and performance, which is more suitable for popularization and application in PAR assembly assistance systems. In conclusion, the developed PAR system can be a valid solution for intelligent manufacturing applications.